[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:42 dist-test-slave-wsx5 krb5kdc[27958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:42 dist-test-slave-wsx5 krb5kdc[27958](info): set up 2 sockets
Sep 23 18:10:42 dist-test-slave-wsx5 krb5kdc[27958](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 18:10:43.483680 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:10:43 dist-test-slave-wsx5 krb5kdc[27958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:10:43.618098 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38477
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:38477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:43.730098 27972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:43.730526 27972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:43.730602 27972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:43.735339 27972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:43.735417 27972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:43.735436 27972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:43.735451 27972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:43.735466 27972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:43.741166 27972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:38477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:43.742436 27972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:43.743500 27972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:43.752628 27977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:43.752628 27980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:43.752704 27972 server_base.cc:1034] running on GCE node
W20260923 18:10:43.752887 27978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:43.753553 27972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:43.754772 27972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:43.755947 27972 hybrid_clock.cc:648] HybridClock initialized: now 1790187043755927 us; error 35 us; skew 500 ppm
Sep 23 18:10:43 dist-test-slave-wsx5 krb5kdc[27958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187043, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:43.758796 27972 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:10:43.759727 27972 webserver.cc:460] Webserver started at http://127.27.75.254:45859/ using document root <none> and password file <none>
I20260923 18:10:43.760262 27972 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:43.760332 27972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:43.760529 27972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:43.762650 27972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "86300bf4c30a42cba259deac90aa51de"
format_stamp: "Formatted at 2026-09-23 18:10:43 on dist-test-slave-wsx5"
server_key: "1ebc6ebf2497b7bbcddac3201c1a287e"
server_key_iv: "74c381c4e86714ceb738eac036b7d1ea"
server_key_version: "encryptionkey@0"
I20260923 18:10:43.763213 27972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "86300bf4c30a42cba259deac90aa51de"
format_stamp: "Formatted at 2026-09-23 18:10:43 on dist-test-slave-wsx5"
server_key: "1ebc6ebf2497b7bbcddac3201c1a287e"
server_key_iv: "74c381c4e86714ceb738eac036b7d1ea"
server_key_version: "encryptionkey@0"
I20260923 18:10:43.767107 27972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:10:43.769814 27987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:43.771200 27972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:43.771337 27972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "86300bf4c30a42cba259deac90aa51de"
format_stamp: "Formatted at 2026-09-23 18:10:43 on dist-test-slave-wsx5"
server_key: "1ebc6ebf2497b7bbcddac3201c1a287e"
server_key_iv: "74c381c4e86714ceb738eac036b7d1ea"
server_key_version: "encryptionkey@0"
I20260923 18:10:43.771450 27972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:43.787601 27972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:43.791203 27972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:43.791409 27972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:43.798338 27972 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:38477
I20260923 18:10:43.798425 28039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:38477 every 8 connection(s)
I20260923 18:10:43.799333 27972 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:43.802250 28040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:43.805235 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 27972
I20260923 18:10:43.805380 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:10:43.806691 27951 external_mini_cluster.cc:1428] Setting key 349644950ebd9d91e7f0e90a36300254
I20260923 18:10:43.808084 28040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de: Bootstrap starting.
I20260923 18:10:43.810916 28040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:43.811728 28040 log.cc:824] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:43.813697 28040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de: No bootstrap required, opened a new log
I20260923 18:10:43.817523 28040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86300bf4c30a42cba259deac90aa51de" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38477 } }
I20260923 18:10:43.817746 28040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:43.817795 28040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86300bf4c30a42cba259deac90aa51de, State: Initialized, Role: FOLLOWER
I20260923 18:10:43.818202 28040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86300bf4c30a42cba259deac90aa51de" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38477 } }
I20260923 18:10:43.818346 28040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:43.818409 28040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:43.818508 28040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:43.819435 28040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86300bf4c30a42cba259deac90aa51de" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38477 } }
I20260923 18:10:43.819816 28040 leader_election.cc:302] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86300bf4c30a42cba259deac90aa51de; no voters:
I20260923 18:10:43.820132 28040 leader_election.cc:288] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:43.820218 28045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:43.820392 28045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [term 1 LEADER]: Becoming Leader. State: Replica: 86300bf4c30a42cba259deac90aa51de, State: Running, Role: LEADER
I20260923 18:10:43.820709 28045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86300bf4c30a42cba259deac90aa51de" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38477 } }
I20260923 18:10:43.821534 28040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:43.822378 28046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86300bf4c30a42cba259deac90aa51de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86300bf4c30a42cba259deac90aa51de" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38477 } } }
I20260923 18:10:43.822533 28046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:43.822841 28047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86300bf4c30a42cba259deac90aa51de. Latest consensus state: current_term: 1 leader_uuid: "86300bf4c30a42cba259deac90aa51de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86300bf4c30a42cba259deac90aa51de" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38477 } } }
I20260923 18:10:43.822927 28047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:43.823268 28054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 23 18:10:43 dist-test-slave-wsx5 krb5kdc[27958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:43.826738 28054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:43.832732 28054 catalog_manager.cc:1347] Generated new cluster ID: 51e294e900cd43f59aac32786ea9e048
I20260923 18:10:43.832794 28054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:43.833246 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:43.809843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58400 (local address 127.27.75.254:38477)
0923 18:10:43.810278 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:43.810293 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:43.811478 (+ 1185us) server_negotiation.cc:407] Connection header received
0923 18:10:43.814913 (+ 3435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:43.814929 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:43.815244 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:43.815542 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:43.817723 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:43.818918 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:43.820350 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:43.820954 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:43.821108 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:43.826138 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:43.826144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:43.826160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:43.826197 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:43.828427 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:43.829315 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:43.829322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:43.829327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:43.829469 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:43.829893 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:43.829896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:43.829898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:43.830430 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:43.830621 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:43.832063 (+ 1442us) server_negotiation.cc:299] Negotiation successful
0923 18:10:43.832291 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260923 18:10:43.845793 28054 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:43.846735 28054 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:43.857728 28054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86300bf4c30a42cba259deac90aa51de: Generated new TSK 0
I20260923 18:10:43.858623 28054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:10:43.896198 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38477
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:44.034490 28066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:44.034768 28066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:44.034813 28066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:44.038828 28066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:44.038892 28066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:44.038951 28066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:10:44.044232 28066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38477
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:44.045248 28066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:44.046204 28066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:44.055356 28071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:44.055306 28074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:44.055405 28066 server_base.cc:1034] running on GCE node
W20260923 18:10:44.055624 28072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:44.056026 28066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:44.056646 28066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:44.057791 28066 hybrid_clock.cc:648] HybridClock initialized: now 1790187044057774 us; error 37 us; skew 500 ppm
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[27958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:44.060551 28066 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:10:44.061545 28066 webserver.cc:460] Webserver started at http://127.27.75.193:35545/ using document root <none> and password file <none>
I20260923 18:10:44.062086 28066 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:44.062143 28066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:44.062352 28066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:44.064409 28066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "324b61c049774982a043f00a94ed0fde"
format_stamp: "Formatted at 2026-09-23 18:10:44 on dist-test-slave-wsx5"
server_key: "95e592f8a5927bae9dc609e0d9452435"
server_key_iv: "bb7c2e28db15cf52cfa2826b1545611f"
server_key_version: "encryptionkey@0"
I20260923 18:10:44.064930 28066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "324b61c049774982a043f00a94ed0fde"
format_stamp: "Formatted at 2026-09-23 18:10:44 on dist-test-slave-wsx5"
server_key: "95e592f8a5927bae9dc609e0d9452435"
server_key_iv: "bb7c2e28db15cf52cfa2826b1545611f"
server_key_version: "encryptionkey@0"
I20260923 18:10:44.068611 28066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:10:44.071221 28081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:44.072610 28066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:10:44.072754 28066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "324b61c049774982a043f00a94ed0fde"
format_stamp: "Formatted at 2026-09-23 18:10:44 on dist-test-slave-wsx5"
server_key: "95e592f8a5927bae9dc609e0d9452435"
server_key_iv: "bb7c2e28db15cf52cfa2826b1545611f"
server_key_version: "encryptionkey@0"
I20260923 18:10:44.072866 28066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:44.088726 28066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:44.092509 28066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:44.092705 28066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:44.093381 28066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:44.094229 28066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:44.094283 28066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:44.094324 28066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:44.094344 28066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:44.102890 28066 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36349
I20260923 18:10:44.103076 28195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36349 every 8 connection(s)
I20260923 18:10:44.103834 28066 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:10:44.113266 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 28066
I20260923 18:10:44.113379 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:10:44.113631 27951 external_mini_cluster.cc:1428] Setting key bfcfb8d28fb85184b7ec23caf36f0e1f
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[27958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:44.118628 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:44.105671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:36709 (local address 127.27.75.254:38477)
0923 18:10:44.105840 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:44.105845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:44.106839 (+ 994us) server_negotiation.cc:407] Connection header received
0923 18:10:44.108117 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:44.108119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:44.108178 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:44.108253 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:44.109669 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.110285 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.111545 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.111852 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.111945 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:44.114566 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:44.114569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:44.114571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:44.114597 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:44.116427 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:44.117015 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:44.117018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:44.117020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:44.117109 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:44.117487 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:44.117491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:44.117493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:44.117686 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:44.117778 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:44.118317 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 18:10:44.118458 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 18:10:44.119563 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:44.106099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38477 (local address 127.27.75.193:36709)
0923 18:10:44.106608 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:44.106657 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:44.107911 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:44.108441 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:44.108448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:44.108894 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:44.109481 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:44.109497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.110428 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.110431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:44.111411 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:44.111421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.112003 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.112006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:44.112163 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:44.112734 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:44.112757 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:44.114394 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:44.116564 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:44.116570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:44.116585 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:44.116907 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:44.117237 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:44.117240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:44.117242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:44.117394 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:44.117792 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:44.117799 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:44.118060 (+ 261us) client_negotiation.cc:769] Sending connection context
0923 18:10:44.118233 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 18:10:44.118575 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":125,"threads_started":1}
I20260923 18:10:44.121081 28196 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38477
I20260923 18:10:44.121376 28196 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:44.121872 28196 heartbeater.cc:511] Master 127.27.75.254:38477 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:10:44.123656 28004 ts_manager.cc:194] Registered new tserver with Master: 324b61c049774982a043f00a94ed0fde (127.27.75.193:36349)
I20260923 18:10:44.124770 28004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:36709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:10:44.144245 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38477
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:44.254969 28201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:44.255231 28201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:44.255293 28201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:44.259296 28201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:44.259363 28201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:44.259442 28201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:10:44.264699 28201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38477
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:44.265795 28201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:44.266845 28201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:44.276306 28206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:44.276312 28207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:44.276360 28201 server_base.cc:1034] running on GCE node
W20260923 18:10:44.276306 28209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:44.277055 28201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:44.277695 28201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:44.278841 28201 hybrid_clock.cc:648] HybridClock initialized: now 1790187044278821 us; error 31 us; skew 500 ppm
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[27958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:44.281395 28201 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:10:44.282214 28201 webserver.cc:460] Webserver started at http://127.27.75.194:44791/ using document root <none> and password file <none>
I20260923 18:10:44.282737 28201 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:44.282799 28201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:44.282980 28201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:44.284919 28201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "633acdb4745648e0ac58bb636fee6312"
format_stamp: "Formatted at 2026-09-23 18:10:44 on dist-test-slave-wsx5"
server_key: "5d867d9a0500e1a1bf660c78ac55d144"
server_key_iv: "36d9b54045103398aad55c6a95ac040b"
server_key_version: "encryptionkey@0"
I20260923 18:10:44.285492 28201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "633acdb4745648e0ac58bb636fee6312"
format_stamp: "Formatted at 2026-09-23 18:10:44 on dist-test-slave-wsx5"
server_key: "5d867d9a0500e1a1bf660c78ac55d144"
server_key_iv: "36d9b54045103398aad55c6a95ac040b"
server_key_version: "encryptionkey@0"
I20260923 18:10:44.289144 28201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:44.291730 28217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:44.293021 28201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:10:44.293221 28201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "633acdb4745648e0ac58bb636fee6312"
format_stamp: "Formatted at 2026-09-23 18:10:44 on dist-test-slave-wsx5"
server_key: "5d867d9a0500e1a1bf660c78ac55d144"
server_key_iv: "36d9b54045103398aad55c6a95ac040b"
server_key_version: "encryptionkey@0"
I20260923 18:10:44.293335 28201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:44.302716 28201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:44.305895 28201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:44.306077 28201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:44.306658 28201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:44.307466 28201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:44.307511 28201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:44.307562 28201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:44.307591 28201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:44.316046 28201 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:35845
I20260923 18:10:44.316067 28330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:35845 every 8 connection(s)
I20260923 18:10:44.316977 28201 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:10:44.320250 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 28201
I20260923 18:10:44.320348 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:10:44.320632 27951 external_mini_cluster.cc:1428] Setting key 77ac57b02f2acb8b954c2652867ffb6e
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[27958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:10:44.331640 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:44.318837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:40981 (local address 127.27.75.254:38477)
0923 18:10:44.319022 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:44.319026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:44.319848 (+ 822us) server_negotiation.cc:407] Connection header received
0923 18:10:44.321357 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:44.321360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:44.321415 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:44.321505 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:44.323039 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.323708 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.324887 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.325219 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.325329 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:44.327889 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:44.327893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:44.327896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:44.327928 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:44.329429 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:44.330025 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:44.330028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:44.330029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:44.330107 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:44.330464 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:44.330466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:44.330467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:44.330655 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:44.330729 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:44.331377 (+ 648us) server_negotiation.cc:299] Negotiation successful
0923 18:10:44.331495 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
I20260923 18:10:44.332577 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:44.319183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38477 (local address 127.27.75.194:40981)
0923 18:10:44.319655 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:44.319735 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:44.321112 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:44.321727 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:44.321742 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:44.322278 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:44.322890 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:44.322908 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.323837 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.323840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:44.324758 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:44.324768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.325419 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.325423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:44.325593 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:44.326259 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:44.326277 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:44.327704 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:44.329580 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:44.329586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:44.329602 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:44.329927 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:44.330231 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:44.330234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:44.330236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:44.330383 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:44.330799 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:44.330804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:44.331085 (+ 281us) client_negotiation.cc:769] Sending connection context
0923 18:10:44.331250 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 18:10:44.331577 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260923 18:10:44.333824 28331 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38477
I20260923 18:10:44.334080 28331 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:44.334605 28331 heartbeater.cc:511] Master 127.27.75.254:38477 requested a full tablet report, sending...
I20260923 18:10:44.335819 28004 ts_manager.cc:194] Registered new tserver with Master: 633acdb4745648e0ac58bb636fee6312 (127.27.75.194:35845)
I20260923 18:10:44.336572 28004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:40981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:10:44.348258 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38477
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:44.460918 28336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:44.461220 28336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:44.461273 28336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:44.465385 28336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:44.465451 28336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:44.465523 28336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:10:44.470870 28336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38477
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:44.471993 28336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:44.473070 28336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:44.483175 28342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:44.483173 28344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:44.483175 28341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:44.483664 28336 server_base.cc:1034] running on GCE node
I20260923 18:10:44.484086 28336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:44.484752 28336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:44.485931 28336 hybrid_clock.cc:648] HybridClock initialized: now 1790187044485897 us; error 52 us; skew 500 ppm
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[27958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:44.488750 28336 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:10:44.489691 28336 webserver.cc:460] Webserver started at http://127.27.75.195:45729/ using document root <none> and password file <none>
I20260923 18:10:44.490254 28336 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:44.490315 28336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:44.490513 28336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:44.492568 28336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "d42cbd12ef2a4de6b3e87e82edc76899"
format_stamp: "Formatted at 2026-09-23 18:10:44 on dist-test-slave-wsx5"
server_key: "f3ae20707f0d14965c62986ed8da60aa"
server_key_iv: "71b5b9631f88b073d38f30eab07f1912"
server_key_version: "encryptionkey@0"
I20260923 18:10:44.493149 28336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "d42cbd12ef2a4de6b3e87e82edc76899"
format_stamp: "Formatted at 2026-09-23 18:10:44 on dist-test-slave-wsx5"
server_key: "f3ae20707f0d14965c62986ed8da60aa"
server_key_iv: "71b5b9631f88b073d38f30eab07f1912"
server_key_version: "encryptionkey@0"
I20260923 18:10:44.497084 28336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:44.499572 28351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:44.500846 28336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:10:44.500958 28336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "d42cbd12ef2a4de6b3e87e82edc76899"
format_stamp: "Formatted at 2026-09-23 18:10:44 on dist-test-slave-wsx5"
server_key: "f3ae20707f0d14965c62986ed8da60aa"
server_key_iv: "71b5b9631f88b073d38f30eab07f1912"
server_key_version: "encryptionkey@0"
I20260923 18:10:44.501071 28336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:44.527292 28336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:44.530486 28336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:44.530663 28336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:44.531289 28336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:44.532133 28336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:44.532179 28336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:44.532220 28336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:44.532243 28336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:44.541257 28336 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:42517
I20260923 18:10:44.541276 28465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:42517 every 8 connection(s)
I20260923 18:10:44.542269 28336 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:10:44.544567 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 28336
I20260923 18:10:44.544672 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:10:44.544981 27951 external_mini_cluster.cc:1428] Setting key d9840a5a55273ebc7648b244f2f04a80
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[27958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:44.555891 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:44.544080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:50675 (local address 127.27.75.254:38477)
0923 18:10:44.544205 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:44.544208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:44.545168 (+ 960us) server_negotiation.cc:407] Connection header received
0923 18:10:44.546732 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:44.546735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:44.546796 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:44.546887 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:44.548191 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.548720 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.549877 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.550097 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.550180 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:44.552436 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:44.552439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:44.552441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:44.552467 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:44.553874 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:44.554388 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:44.554391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:44.554392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:44.554476 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:44.554806 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:44.554809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:44.554810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:44.555012 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:44.555102 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:44.555638 (+ 536us) server_negotiation.cc:299] Negotiation successful
0923 18:10:44.555765 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 18:10:44.556754 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:44.544461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38477 (local address 127.27.75.195:50675)
0923 18:10:44.544984 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:44.545042 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:44.546445 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:44.547065 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:44.547071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:44.547499 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:44.548047 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:44.548060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.548817 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.548819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:44.549759 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:44.549768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.550219 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.550222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:44.550354 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:44.550861 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:44.550907 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:44.552271 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:44.554003 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:44.554008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:44.554022 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:44.554291 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:44.554554 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:44.554556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:44.554557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:44.554700 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:44.555124 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:44.555129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:44.555389 (+ 260us) client_negotiation.cc:769] Sending connection context
0923 18:10:44.555554 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 18:10:44.555834 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260923 18:10:44.558384 28466 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38477
I20260923 18:10:44.558687 28466 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:44.559310 28466 heartbeater.cc:511] Master 127.27.75.254:38477 requested a full tablet report, sending...
I20260923 18:10:44.560493 28004 ts_manager.cc:194] Registered new tserver with Master: d42cbd12ef2a4de6b3e87e82edc76899 (127.27.75.195:42517)
I20260923 18:10:44.561102 28004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:50675
I20260923 18:10:44.571880 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[27958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187044, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[27958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187044, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:10:44.597724 28474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:44.588090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.27.75.193:36349)
0923 18:10:44.588416 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:44.588430 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:44.588455 (+ 25us) server_negotiation.cc:407] Connection header received
0923 18:10:44.588576 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:44.588581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:44.588771 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:44.588925 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:44.589676 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.590513 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.591473 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.591880 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.591968 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:44.594477 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:44.594486 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:44.594500 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:44.594528 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:44.596016 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:44.596460 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:44.596466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:44.596468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:44.596549 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:44.596815 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:44.596817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:44.596818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:44.597049 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:44.597232 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:44.597419 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 18:10:44.597561 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":141,"threads_started":1}
W20260923 18:10:44.598693 28109 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33398
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[27958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[27958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:10:44.623819 28474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:44.615562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.27.75.193:36349)
0923 18:10:44.615698 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:44.615702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:44.615844 (+ 142us) server_negotiation.cc:407] Connection header received
0923 18:10:44.615989 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:44.615991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:44.616037 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:44.616107 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:44.616832 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.617424 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.618269 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:44.618499 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:44.618572 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:44.620834 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:44.620837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:44.620839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:44.620862 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:44.622229 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:44.622635 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:44.622638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:44.622639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:44.622725 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:44.623047 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:44.623049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:44.623050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:44.623208 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:44.623301 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:44.623580 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 18:10:44.623696 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 18:10:44.625164 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 28066
I20260923 18:10:44.630409 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 28201
I20260923 18:10:44.634968 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 28336
I20260923 18:10:44.639967 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 27972
2026-09-23T18:10:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2144 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[28481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[28481](info): set up 2 sockets
Sep 23 18:10:44 dist-test-slave-wsx5 krb5kdc[28481](info): commencing operation
krb5kdc: starting...
W20260923 18:10:45.598858 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:10:45 dist-test-slave-wsx5 krb5kdc[28481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:10:45.718755 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:41251
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:41251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:45.833195 28494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:45.833475 28494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:45.833516 28494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:45.837494 28494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:45.837561 28494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:45.837580 28494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:45.837595 28494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:45.837610 28494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:45.842903 28494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:41251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:41251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:45.843920 28494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:45.844928 28494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:45.853446 28502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:45.853510 28499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:45.853499 28500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:45.853843 28494 server_base.cc:1034] running on GCE node
I20260923 18:10:45.854457 28494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:45.855585 28494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:45.856771 28494 hybrid_clock.cc:648] HybridClock initialized: now 1790187045856744 us; error 42 us; skew 500 ppm
Sep 23 18:10:45 dist-test-slave-wsx5 krb5kdc[28481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187045, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:45.859567 28494 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:10:45.860512 28494 webserver.cc:460] Webserver started at http://127.27.75.254:34009/ using document root <none> and password file <none>
I20260923 18:10:45.861029 28494 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:45.861088 28494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:45.861308 28494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:45.863334 28494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "44525eb4aad4473aa05303a6b2b30b19"
format_stamp: "Formatted at 2026-09-23 18:10:45 on dist-test-slave-wsx5"
server_key: "e579bdfd18dca741dee157da02244434"
server_key_iv: "88248606c1122f37239ad078e5ffc14f"
server_key_version: "encryptionkey@0"
I20260923 18:10:45.863852 28494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "44525eb4aad4473aa05303a6b2b30b19"
format_stamp: "Formatted at 2026-09-23 18:10:45 on dist-test-slave-wsx5"
server_key: "e579bdfd18dca741dee157da02244434"
server_key_iv: "88248606c1122f37239ad078e5ffc14f"
server_key_version: "encryptionkey@0"
I20260923 18:10:45.867833 28494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 18:10:45.870615 28509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:45.872069 28494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:10:45.872195 28494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "44525eb4aad4473aa05303a6b2b30b19"
format_stamp: "Formatted at 2026-09-23 18:10:45 on dist-test-slave-wsx5"
server_key: "e579bdfd18dca741dee157da02244434"
server_key_iv: "88248606c1122f37239ad078e5ffc14f"
server_key_version: "encryptionkey@0"
I20260923 18:10:45.872308 28494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:45.895676 28494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:45.899037 28494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:45.899206 28494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:45.906467 28494 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:41251
I20260923 18:10:45.906550 28561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:41251 every 8 connection(s)
I20260923 18:10:45.907510 28494 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:45.910460 28562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:45.915385 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 28494
I20260923 18:10:45.915509 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:10:45.915768 27951 external_mini_cluster.cc:1428] Setting key cf5397d732f68d6bf4cb7df0280e6e1e
I20260923 18:10:45.916566 28562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19: Bootstrap starting.
I20260923 18:10:45.919003 28562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:45.919943 28562 log.cc:824] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:45.922291 28562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19: No bootstrap required, opened a new log
Sep 23 18:10:45 dist-test-slave-wsx5 krb5kdc[28481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:45.926512 28562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44525eb4aad4473aa05303a6b2b30b19" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41251 } }
I20260923 18:10:45.926769 28562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:45.926828 28562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44525eb4aad4473aa05303a6b2b30b19, State: Initialized, Role: FOLLOWER
I20260923 18:10:45.927309 28562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44525eb4aad4473aa05303a6b2b30b19" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41251 } }
I20260923 18:10:45.927461 28562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:45.927526 28562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:45.927627 28562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:45.928524 28562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44525eb4aad4473aa05303a6b2b30b19" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41251 } }
I20260923 18:10:45.928867 28562 leader_election.cc:302] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44525eb4aad4473aa05303a6b2b30b19; no voters:
I20260923 18:10:45.929194 28562 leader_election.cc:288] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:45.929333 28567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:45.929502 28567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [term 1 LEADER]: Becoming Leader. State: Replica: 44525eb4aad4473aa05303a6b2b30b19, State: Running, Role: LEADER
I20260923 18:10:45.929874 28567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44525eb4aad4473aa05303a6b2b30b19" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41251 } }
I20260923 18:10:45.930594 28562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:45.931104 28567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44525eb4aad4473aa05303a6b2b30b19. Latest consensus state: current_term: 1 leader_uuid: "44525eb4aad4473aa05303a6b2b30b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44525eb4aad4473aa05303a6b2b30b19" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41251 } } }
I20260923 18:10:45.931268 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:45.917232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.27.75.254:41251)
0923 18:10:45.917652 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:45.917669 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:45.917752 (+ 83us) server_negotiation.cc:407] Connection header received
0923 18:10:45.918419 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:45.918437 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:45.918794 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:45.919086 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:45.919983 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:45.921293 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:45.922450 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:45.922942 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:45.923067 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:45.925844 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:45.925849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:45.925865 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:45.925895 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:45.927720 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:45.928200 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:45.928206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:45.928212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:45.928333 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:45.928610 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:45.928613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:45.928615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:45.929084 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:45.929295 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:45.930071 (+ 776us) server_negotiation.cc:299] Negotiation successful
0923 18:10:45.930340 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260923 18:10:45.931315 28567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:45.931491 28568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44525eb4aad4473aa05303a6b2b30b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44525eb4aad4473aa05303a6b2b30b19" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41251 } } }
I20260923 18:10:45.931599 28568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:45.931910 28576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:45.935287 28576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:45.941843 28576 catalog_manager.cc:1347] Generated new cluster ID: 4322e190a636476abb980aadb35a2aaa
I20260923 18:10:45.941924 28576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:45.964414 28576 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:45.965529 28576 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:45.973733 28576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44525eb4aad4473aa05303a6b2b30b19: Generated new TSK 0
I20260923 18:10:45.974658 28576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:10:46.008777 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41251
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:46.120091 28588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:46.120352 28588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:46.120396 28588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:46.124439 28588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:46.124511 28588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:46.124572 28588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:10:46.129890 28588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41251
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:46.131012 28588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:46.132133 28588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:46.142233 28593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:46.142470 28588 server_base.cc:1034] running on GCE node
W20260923 18:10:46.142236 28596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:46.142256 28594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:46.142977 28588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:46.143612 28588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:46.144780 28588 hybrid_clock.cc:648] HybridClock initialized: now 1790187046144751 us; error 47 us; skew 500 ppm
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[28481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187046, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:46.147469 28588 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:10:46.148303 28588 webserver.cc:460] Webserver started at http://127.27.75.193:35301/ using document root <none> and password file <none>
I20260923 18:10:46.148856 28588 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:46.148924 28588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:46.149144 28588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:46.151374 28588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "26b67593c27643c3a0d84e3ba362a2f9"
format_stamp: "Formatted at 2026-09-23 18:10:46 on dist-test-slave-wsx5"
server_key: "2096244ce222e23e2b250dfcf999c833"
server_key_iv: "10e2cfdab379e534088792bee9c4429e"
server_key_version: "encryptionkey@0"
I20260923 18:10:46.151909 28588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "26b67593c27643c3a0d84e3ba362a2f9"
format_stamp: "Formatted at 2026-09-23 18:10:46 on dist-test-slave-wsx5"
server_key: "2096244ce222e23e2b250dfcf999c833"
server_key_iv: "10e2cfdab379e534088792bee9c4429e"
server_key_version: "encryptionkey@0"
I20260923 18:10:46.155606 28588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 18:10:46.158038 28603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:46.159281 28588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:10:46.159416 28588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "26b67593c27643c3a0d84e3ba362a2f9"
format_stamp: "Formatted at 2026-09-23 18:10:46 on dist-test-slave-wsx5"
server_key: "2096244ce222e23e2b250dfcf999c833"
server_key_iv: "10e2cfdab379e534088792bee9c4429e"
server_key_version: "encryptionkey@0"
I20260923 18:10:46.159513 28588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:46.184144 28588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:46.187311 28588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:46.187493 28588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:46.188071 28588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:46.188908 28588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:46.188956 28588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:46.189009 28588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:46.189034 28588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:46.197378 28588 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36721
I20260923 18:10:46.197413 28717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36721 every 8 connection(s)
I20260923 18:10:46.198244 28588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:10:46.205195 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 28588
I20260923 18:10:46.205307 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:10:46.205561 27951 external_mini_cluster.cc:1428] Setting key 0abc0e66c808c814010f27d6d3b3e219
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[28481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187046, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:46.212116 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.200132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:40181 (local address 127.27.75.254:41251)
0923 18:10:46.200282 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.200286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.201099 (+ 813us) server_negotiation.cc:407] Connection header received
0923 18:10:46.202468 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.202471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.202528 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.202600 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:46.203757 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.204291 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.205682 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.205934 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.206046 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.208766 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:46.208769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:46.208771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:46.208796 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:46.210125 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.210620 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.210623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.210624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.210707 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.211028 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.211031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.211033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.211215 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:46.211299 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.211797 (+ 498us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.211938 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 18:10:46.212929 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.200474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:41251 (local address 127.27.75.193:40181)
0923 18:10:46.200948 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:46.200993 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:46.202264 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:46.202774 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:46.202781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:46.203126 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:46.203611 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.203624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.204408 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.204411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.205531 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.205542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.206109 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.206114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.206342 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.206998 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:46.207015 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:46.208555 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:46.210240 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:46.210246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:46.210262 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:46.210524 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:46.210796 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:46.210799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:46.210801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:46.210941 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:46.211310 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:46.211324 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:46.211571 (+ 247us) client_negotiation.cc:769] Sending connection context
0923 18:10:46.211744 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 18:10:46.212078 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:10:46.214267 28718 heartbeater.cc:347] Connected to a master server at 127.27.75.254:41251
I20260923 18:10:46.214497 28718 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:46.215027 28718 heartbeater.cc:511] Master 127.27.75.254:41251 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:10:46.216708 28526 ts_manager.cc:194] Registered new tserver with Master: 26b67593c27643c3a0d84e3ba362a2f9 (127.27.75.193:36721)
I20260923 18:10:46.218194 28526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:40181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:10:46.232896 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41251
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:46.344276 28724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:46.344574 28724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:46.344636 28724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:46.348814 28724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:46.348891 28724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:46.348976 28724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:10:46.354676 28724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41251
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:46.355799 28724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:46.356861 28724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:46.367609 28729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:46.367617 28730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:46.367627 28732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:46.367756 28724 server_base.cc:1034] running on GCE node
I20260923 18:10:46.368399 28724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:46.369371 28724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:46.370586 28724 hybrid_clock.cc:648] HybridClock initialized: now 1790187046370555 us; error 41 us; skew 500 ppm
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[28481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187046, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:46.373582 28724 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:10:46.374571 28724 webserver.cc:460] Webserver started at http://127.27.75.194:41761/ using document root <none> and password file <none>
I20260923 18:10:46.375130 28724 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:46.375202 28724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:46.375398 28724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:46.377600 28724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "ca9892a4b82b42ed9b83846788d5b90f"
format_stamp: "Formatted at 2026-09-23 18:10:46 on dist-test-slave-wsx5"
server_key: "300e21446a600d256b0cdbaa74fee3d1"
server_key_iv: "3ca5611cb2399cb5de952f0eaee8605a"
server_key_version: "encryptionkey@0"
I20260923 18:10:46.378187 28724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "ca9892a4b82b42ed9b83846788d5b90f"
format_stamp: "Formatted at 2026-09-23 18:10:46 on dist-test-slave-wsx5"
server_key: "300e21446a600d256b0cdbaa74fee3d1"
server_key_iv: "3ca5611cb2399cb5de952f0eaee8605a"
server_key_version: "encryptionkey@0"
I20260923 18:10:46.382190 28724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:10:46.384936 28739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:46.386432 28724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:46.386564 28724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "ca9892a4b82b42ed9b83846788d5b90f"
format_stamp: "Formatted at 2026-09-23 18:10:46 on dist-test-slave-wsx5"
server_key: "300e21446a600d256b0cdbaa74fee3d1"
server_key_iv: "3ca5611cb2399cb5de952f0eaee8605a"
server_key_version: "encryptionkey@0"
I20260923 18:10:46.386682 28724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:46.407389 28724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:46.411072 28724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:46.411279 28724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:46.411924 28724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:46.412861 28724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:46.412910 28724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:46.412966 28724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:46.412993 28724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:46.421598 28724 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:33717
I20260923 18:10:46.421622 28852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:33717 every 8 connection(s)
I20260923 18:10:46.422580 28724 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:10:46.429798 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 28724
I20260923 18:10:46.429901 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:10:46.430160 27951 external_mini_cluster.cc:1428] Setting key 1a240b6e404a270f4126f1805ed4c9fb
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[28481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187046, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:46.437148 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.424480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:42511 (local address 127.27.75.254:41251)
0923 18:10:46.424632 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.424635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.425558 (+ 923us) server_negotiation.cc:407] Connection header received
0923 18:10:46.426858 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.426861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.426917 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.426998 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:46.428191 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.428772 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.430019 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.430368 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.430437 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.433368 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:46.433372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:46.433375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:46.433401 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:46.434983 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.435552 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.435556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.435558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.435639 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.435982 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.435986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.435988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.436182 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:46.436296 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.436824 (+ 528us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.436954 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 18:10:46.438036 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.424860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:41251 (local address 127.27.75.194:42511)
0923 18:10:46.425389 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:46.425434 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:46.426614 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:46.427151 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:46.427158 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:46.427544 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:46.428031 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.428044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.428886 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.428889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.429865 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.429877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.430628 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.430632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.430827 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.431476 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:46.431498 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:46.433174 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:46.435120 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:46.435127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:46.435146 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:46.435442 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:46.435726 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:46.435729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:46.435731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:46.435879 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:46.436307 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:46.436314 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:46.436576 (+ 262us) client_negotiation.cc:769] Sending connection context
0923 18:10:46.436752 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 18:10:46.437111 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:10:46.439566 28853 heartbeater.cc:347] Connected to a master server at 127.27.75.254:41251
I20260923 18:10:46.439842 28853 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:46.440438 28853 heartbeater.cc:511] Master 127.27.75.254:41251 requested a full tablet report, sending...
I20260923 18:10:46.441812 28526 ts_manager.cc:194] Registered new tserver with Master: ca9892a4b82b42ed9b83846788d5b90f (127.27.75.194:33717)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
I20260923 18:10:46.442600 28526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:42511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:10:46.457866 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41251
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:46.570996 28858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:46.571291 28858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:46.571339 28858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:46.575706 28858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:46.575795 28858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:46.575875 28858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:10:46.581509 28858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41251
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:46.582589 28858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:46.583717 28858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:46.593909 28863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:46.594208 28864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:46.594195 28866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:46.594473 28858 server_base.cc:1034] running on GCE node
I20260923 18:10:46.594941 28858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:46.595659 28858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:46.596859 28858 hybrid_clock.cc:648] HybridClock initialized: now 1790187046596845 us; error 31 us; skew 500 ppm
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[28481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187046, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:46.599985 28858 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:10:46.600976 28858 webserver.cc:460] Webserver started at http://127.27.75.195:41165/ using document root <none> and password file <none>
I20260923 18:10:46.601609 28858 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:46.601676 28858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:46.601883 28858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:46.604085 28858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "8c563a9ad5eb494e882081980509aec9"
format_stamp: "Formatted at 2026-09-23 18:10:46 on dist-test-slave-wsx5"
server_key: "3aa7074f526d38136d8c95afca5438c4"
server_key_iv: "a4d1d5a32a7e9a3ff6b83fb252a263ff"
server_key_version: "encryptionkey@0"
I20260923 18:10:46.604657 28858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "8c563a9ad5eb494e882081980509aec9"
format_stamp: "Formatted at 2026-09-23 18:10:46 on dist-test-slave-wsx5"
server_key: "3aa7074f526d38136d8c95afca5438c4"
server_key_iv: "a4d1d5a32a7e9a3ff6b83fb252a263ff"
server_key_version: "encryptionkey@0"
I20260923 18:10:46.608909 28858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 18:10:46.611557 28873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:46.612972 28858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:10:46.613111 28858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "8c563a9ad5eb494e882081980509aec9"
format_stamp: "Formatted at 2026-09-23 18:10:46 on dist-test-slave-wsx5"
server_key: "3aa7074f526d38136d8c95afca5438c4"
server_key_iv: "a4d1d5a32a7e9a3ff6b83fb252a263ff"
server_key_version: "encryptionkey@0"
I20260923 18:10:46.613265 28858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:46.646469 28858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:46.649675 28858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:46.649868 28858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:46.650444 28858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:46.651309 28858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:46.651357 28858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:46.651397 28858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:46.651412 28858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:46.659623 28858 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:37753
I20260923 18:10:46.659682 28988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:37753 every 8 connection(s)
I20260923 18:10:46.660513 28858 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:10:46.664847 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 28858
I20260923 18:10:46.664970 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:10:46.665292 27951 external_mini_cluster.cc:1428] Setting key 108d2d657847123947a6bf85e07e12ee
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[28481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187046, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:46.675158 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.662506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:39287 (local address 127.27.75.254:41251)
0923 18:10:46.662628 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.662632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.663591 (+ 959us) server_negotiation.cc:407] Connection header received
0923 18:10:46.664989 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.664993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.665057 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.665155 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:46.666603 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.667205 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.668428 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.668659 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.668747 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.671339 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:46.671342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:46.671344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:46.671373 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:46.672829 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.673474 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.673478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.673479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.673569 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.673999 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.674002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.674003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.674219 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:46.674300 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.674898 (+ 598us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.675019 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 18:10:46.675984 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.662889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:41251 (local address 127.27.75.195:39287)
0923 18:10:46.663434 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:46.663484 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:46.664749 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:46.665366 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:46.665377 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:46.665795 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:46.666447 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.666464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.667337 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.667339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.668302 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.668312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.668803 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.668807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.668978 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.669574 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:46.669591 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:46.671147 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:46.672963 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:46.672970 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:46.672986 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:46.673328 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:46.673705 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:46.673708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:46.673710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:46.673877 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:46.674340 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:46.674346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:46.674612 (+ 266us) client_negotiation.cc:769] Sending connection context
0923 18:10:46.674788 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 18:10:46.675109 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":137,"threads_started":1}
I20260923 18:10:46.677304 28989 heartbeater.cc:347] Connected to a master server at 127.27.75.254:41251
I20260923 18:10:46.677561 28989 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:46.678095 28989 heartbeater.cc:511] Master 127.27.75.254:41251 requested a full tablet report, sending...
I20260923 18:10:46.679210 28526 ts_manager.cc:194] Registered new tserver with Master: 8c563a9ad5eb494e882081980509aec9 (127.27.75.195:37753)
I20260923 18:10:46.679728 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:10:46.679776 28526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:39287
I20260923 18:10:46.690531 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.683183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.27.75.254:41251)
0923 18:10:46.683304 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.683307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.683478 (+ 171us) server_negotiation.cc:407] Connection header received
0923 18:10:46.683577 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.683580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.683618 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.683684 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:46.684305 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.684847 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.685799 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.686017 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.686105 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.687468 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:46.687471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:46.687473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:46.687497 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:46.688759 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.689208 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.689211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.689212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.689290 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.689623 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.689626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.689629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.689855 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:46.689971 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.690281 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.690398 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 18:10:46.695708 28526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:10:46.698771 28526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:46.713328 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.707811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:33717 (local address 127.0.0.1:57380)
0923 18:10:46.708132 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:46.708184 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:46.708315 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:46.709034 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:46.709041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:46.709065 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:46.709251 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.709258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.710541 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.710544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.711613 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.711622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.712783 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.712786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.712869 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.712904 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 18:10:46.713013 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 18:10:46.713161 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
I20260923 18:10:46.713483 29004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.708673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.27.75.193:36721)
0923 18:10:46.709367 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.709381 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.709397 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:10:46.709467 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.709472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.709613 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.709791 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:46.710002 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.710956 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.712012 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.712828 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.712929 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.713057 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.713236 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.713334 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":106,"threads_started":1}
I20260923 18:10:46.713836 29001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.707980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.27.75.194:33717)
0923 18:10:46.708341 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.708354 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.708387 (+ 33us) server_negotiation.cc:407] Connection header received
0923 18:10:46.708461 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.708466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.708677 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.708834 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:46.709384 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.710410 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.711738 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.712523 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.712618 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.712733 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.713581 (+ 848us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.713702 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":151,"threads_started":1}
I20260923 18:10:46.713331 29003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.708465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36721 (local address 127.0.0.1:52184)
0923 18:10:46.708758 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:46.708771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:46.708874 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:46.709770 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:46.709774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:46.709791 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:46.709880 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.709886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.711094 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.711097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.711884 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.711894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.712948 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.712952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.713083 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.713103 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 18:10:46.713182 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 18:10:46.713235 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":23,"thread_start_us":100,"threads_started":1}
I20260923 18:10:46.715118 29002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.708536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:37753 (local address 127.0.0.1:59500)
0923 18:10:46.709038 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:46.709049 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:46.709154 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:46.709948 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:46.709951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:46.709969 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:46.710065 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.710071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.711927 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.711930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.713048 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.713057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.714747 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.714750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.714818 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.714838 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 18:10:46.714915 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 18:10:46.714989 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260923 18:10:46.715188 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.708882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.27.75.195:37753)
0923 18:10:46.709308 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.709321 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.709343 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:10:46.709414 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.709420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.709606 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.709791 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:46.710182 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.710972 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.713186 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.714006 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.714140 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.714277 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.714965 (+ 688us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.715060 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":96,"threads_started":1}
I20260923 18:10:46.716974 28652 tablet_service.cc:1467] Processing CreateTablet for tablet 0c66ffb225b94f1797e7dc419e503d10 (DEFAULT_TABLE table=test-table [id=fb81dab477684168963552d9ee1e1b7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:46.716974 28787 tablet_service.cc:1467] Processing CreateTablet for tablet 0c66ffb225b94f1797e7dc419e503d10 (DEFAULT_TABLE table=test-table [id=fb81dab477684168963552d9ee1e1b7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:46.717830 28922 tablet_service.cc:1467] Processing CreateTablet for tablet 0c66ffb225b94f1797e7dc419e503d10 (DEFAULT_TABLE table=test-table [id=fb81dab477684168963552d9ee1e1b7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:46.718261 28652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c66ffb225b94f1797e7dc419e503d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:46.718261 28787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c66ffb225b94f1797e7dc419e503d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:46.718734 28922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c66ffb225b94f1797e7dc419e503d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:46.724928 29006 tablet_bootstrap.cc:492] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9: Bootstrap starting.
I20260923 18:10:46.727005 29008 tablet_bootstrap.cc:492] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f: Bootstrap starting.
I20260923 18:10:46.727388 29007 tablet_bootstrap.cc:492] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9: Bootstrap starting.
I20260923 18:10:46.729364 29006 tablet_bootstrap.cc:654] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:46.730476 29006 log.cc:824] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:46.732686 29008 tablet_bootstrap.cc:654] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:46.733233 29006 tablet_bootstrap.cc:492] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9: No bootstrap required, opened a new log
I20260923 18:10:46.733446 29006 ts_tablet_manager.cc:1397] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 18:10:46.734591 29007 tablet_bootstrap.cc:654] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:46.735508 29008 log.cc:824] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:46.735548 29007 log.cc:824] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:46.737560 29006 raft_consensus.cc:348] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9892a4b82b42ed9b83846788d5b90f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 33717 } } peers { permanent_uuid: "8c563a9ad5eb494e882081980509aec9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 37753 } } peers { permanent_uuid: "26b67593c27643c3a0d84e3ba362a2f9" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36721 } }
I20260923 18:10:46.738014 29007 tablet_bootstrap.cc:492] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9: No bootstrap required, opened a new log
I20260923 18:10:46.738040 29006 raft_consensus.cc:374] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:46.738129 29006 raft_consensus.cc:729] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c563a9ad5eb494e882081980509aec9, State: Initialized, Role: FOLLOWER
I20260923 18:10:46.738346 29007 ts_tablet_manager.cc:1397] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260923 18:10:46.738611 29008 tablet_bootstrap.cc:492] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f: No bootstrap required, opened a new log
I20260923 18:10:46.738739 29006 consensus_queue.cc:260] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9892a4b82b42ed9b83846788d5b90f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 33717 } } peers { permanent_uuid: "8c563a9ad5eb494e882081980509aec9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 37753 } } peers { permanent_uuid: "26b67593c27643c3a0d84e3ba362a2f9" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36721 } }
I20260923 18:10:46.738859 29008 ts_tablet_manager.cc:1397] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260923 18:10:46.739596 28989 heartbeater.cc:503] Master 127.27.75.254:41251 was elected leader, sending a full tablet report...
I20260923 18:10:46.740003 29006 ts_tablet_manager.cc:1428] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 18:10:46.743469 29008 raft_consensus.cc:348] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9892a4b82b42ed9b83846788d5b90f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 33717 } } peers { permanent_uuid: "8c563a9ad5eb494e882081980509aec9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 37753 } } peers { permanent_uuid: "26b67593c27643c3a0d84e3ba362a2f9" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36721 } }
I20260923 18:10:46.743798 29008 raft_consensus.cc:374] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:46.743863 29008 raft_consensus.cc:729] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9892a4b82b42ed9b83846788d5b90f, State: Initialized, Role: FOLLOWER
I20260923 18:10:46.744330 29008 consensus_queue.cc:260] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9892a4b82b42ed9b83846788d5b90f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 33717 } } peers { permanent_uuid: "8c563a9ad5eb494e882081980509aec9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 37753 } } peers { permanent_uuid: "26b67593c27643c3a0d84e3ba362a2f9" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36721 } }
I20260923 18:10:46.744448 29007 raft_consensus.cc:348] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9892a4b82b42ed9b83846788d5b90f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 33717 } } peers { permanent_uuid: "8c563a9ad5eb494e882081980509aec9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 37753 } } peers { permanent_uuid: "26b67593c27643c3a0d84e3ba362a2f9" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36721 } }
I20260923 18:10:46.744989 28853 heartbeater.cc:503] Master 127.27.75.254:41251 was elected leader, sending a full tablet report...
I20260923 18:10:46.745285 29007 raft_consensus.cc:374] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:46.745352 29007 raft_consensus.cc:729] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b67593c27643c3a0d84e3ba362a2f9, State: Initialized, Role: FOLLOWER
I20260923 18:10:46.745427 29008 ts_tablet_manager.cc:1428] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260923 18:10:46.745811 29007 consensus_queue.cc:260] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9892a4b82b42ed9b83846788d5b90f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 33717 } } peers { permanent_uuid: "8c563a9ad5eb494e882081980509aec9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 37753 } } peers { permanent_uuid: "26b67593c27643c3a0d84e3ba362a2f9" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36721 } }
I20260923 18:10:46.746600 28718 heartbeater.cc:503] Master 127.27.75.254:41251 was elected leader, sending a full tablet report...
I20260923 18:10:46.747802 29007 ts_tablet_manager.cc:1428] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260923 18:10:46.787429 29014 raft_consensus.cc:482] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:46.787652 29014 raft_consensus.cc:504] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9892a4b82b42ed9b83846788d5b90f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 33717 } } peers { permanent_uuid: "8c563a9ad5eb494e882081980509aec9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 37753 } } peers { permanent_uuid: "26b67593c27643c3a0d84e3ba362a2f9" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36721 } }
I20260923 18:10:46.788808 29014 leader_election.cc:288] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca9892a4b82b42ed9b83846788d5b90f (127.27.75.194:33717), 8c563a9ad5eb494e882081980509aec9 (127.27.75.195:37753)
I20260923 18:10:46.793242 29001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.789202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:42833 (local address 127.27.75.194:33717)
0923 18:10:46.789415 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.789419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.789563 (+ 144us) server_negotiation.cc:407] Connection header received
0923 18:10:46.789782 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.789785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.789840 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.789891 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:46.790334 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.791010 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.792255 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.792686 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.792789 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.792843 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.793010 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.793087 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 18:10:46.793283 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.789703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:37033 (local address 127.27.75.195:37753)
0923 18:10:46.789898 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.789902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.790051 (+ 149us) server_negotiation.cc:407] Connection header received
0923 18:10:46.790153 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.790156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.790198 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.790271 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:46.790622 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.791169 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.792345 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.792745 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.792857 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.792906 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.793074 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.793180 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260923 18:10:46.793285 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.789089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:33717 (local address 127.27.75.193:42833)
0923 18:10:46.789463 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:46.789477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:46.789650 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:46.790033 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:46.790036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:46.790058 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:46.790178 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.790184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.791118 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.791121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.792125 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.792135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.792796 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.792800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.792891 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.792913 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:10:46.792982 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 18:10:46.793053 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":32,"thread_start_us":131,"threads_started":1}
I20260923 18:10:46.793293 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.789576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:37753 (local address 127.27.75.193:37033)
0923 18:10:46.789961 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:46.789974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:46.790076 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:46.790356 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:46.790359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:46.790380 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:46.790500 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.790527 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.791314 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.791318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.792217 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:46.792228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.792860 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.792864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:46.792954 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.792975 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:10:46.793050 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 18:10:46.793113 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260923 18:10:46.794005 28807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c66ffb225b94f1797e7dc419e503d10" candidate_uuid: "26b67593c27643c3a0d84e3ba362a2f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9892a4b82b42ed9b83846788d5b90f" is_pre_election: true
I20260923 18:10:46.794008 28942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c66ffb225b94f1797e7dc419e503d10" candidate_uuid: "26b67593c27643c3a0d84e3ba362a2f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c563a9ad5eb494e882081980509aec9" is_pre_election: true
I20260923 18:10:46.794291 28942 raft_consensus.cc:2454] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b67593c27643c3a0d84e3ba362a2f9 in term 0.
I20260923 18:10:46.794322 28807 raft_consensus.cc:2454] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b67593c27643c3a0d84e3ba362a2f9 in term 0.
I20260923 18:10:46.794765 28606 leader_election.cc:302] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b67593c27643c3a0d84e3ba362a2f9, 8c563a9ad5eb494e882081980509aec9; no voters:
I20260923 18:10:46.795053 29014 raft_consensus.cc:2781] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:46.795131 29014 raft_consensus.cc:482] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:46.795162 29014 raft_consensus.cc:3037] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:46.796205 29014 raft_consensus.cc:504] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9892a4b82b42ed9b83846788d5b90f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 33717 } } peers { permanent_uuid: "8c563a9ad5eb494e882081980509aec9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 37753 } } peers { permanent_uuid: "26b67593c27643c3a0d84e3ba362a2f9" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36721 } }
I20260923 18:10:46.796545 29014 leader_election.cc:288] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [CANDIDATE]: Term 1 election: Requested vote from peers ca9892a4b82b42ed9b83846788d5b90f (127.27.75.194:33717), 8c563a9ad5eb494e882081980509aec9 (127.27.75.195:37753)
I20260923 18:10:46.796980 28807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c66ffb225b94f1797e7dc419e503d10" candidate_uuid: "26b67593c27643c3a0d84e3ba362a2f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9892a4b82b42ed9b83846788d5b90f"
I20260923 18:10:46.797108 28807 raft_consensus.cc:3037] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:46.797150 28942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c66ffb225b94f1797e7dc419e503d10" candidate_uuid: "26b67593c27643c3a0d84e3ba362a2f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c563a9ad5eb494e882081980509aec9"
I20260923 18:10:46.797272 28942 raft_consensus.cc:3037] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:46.798249 28807 raft_consensus.cc:2454] T 0c66ffb225b94f1797e7dc419e503d10 P ca9892a4b82b42ed9b83846788d5b90f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b67593c27643c3a0d84e3ba362a2f9 in term 1.
I20260923 18:10:46.798327 28942 raft_consensus.cc:2454] T 0c66ffb225b94f1797e7dc419e503d10 P 8c563a9ad5eb494e882081980509aec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b67593c27643c3a0d84e3ba362a2f9 in term 1.
I20260923 18:10:46.798574 28606 leader_election.cc:302] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b67593c27643c3a0d84e3ba362a2f9, ca9892a4b82b42ed9b83846788d5b90f; no voters:
I20260923 18:10:46.798811 29014 raft_consensus.cc:2781] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:46.799037 29014 raft_consensus.cc:686] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [term 1 LEADER]: Becoming Leader. State: Replica: 26b67593c27643c3a0d84e3ba362a2f9, State: Running, Role: LEADER
I20260923 18:10:46.799437 29014 consensus_queue.cc:237] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9892a4b82b42ed9b83846788d5b90f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 33717 } } peers { permanent_uuid: "8c563a9ad5eb494e882081980509aec9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 37753 } } peers { permanent_uuid: "26b67593c27643c3a0d84e3ba362a2f9" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36721 } }
I20260923 18:10:46.802802 28521 catalog_manager.cc:5495] T 0c66ffb225b94f1797e7dc419e503d10 P 26b67593c27643c3a0d84e3ba362a2f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b67593c27643c3a0d84e3ba362a2f9 (127.27.75.193). New cstate: current_term: 1 leader_uuid: "26b67593c27643c3a0d84e3ba362a2f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9892a4b82b42ed9b83846788d5b90f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c563a9ad5eb494e882081980509aec9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 37753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b67593c27643c3a0d84e3ba362a2f9" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36721 } health_report { overall_health: HEALTHY } } }
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[28481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:10:46.829320 29004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.820311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.27.75.193:36721)
0923 18:10:46.820481 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.820485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.820579 (+ 94us) server_negotiation.cc:407] Connection header received
0923 18:10:46.820786 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.820790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.820840 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.820906 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:46.821665 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.822203 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.823058 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.823298 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.823387 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.825966 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:46.825979 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:46.825994 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:46.826021 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:46.827553 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.828072 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.828078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.828080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.828164 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.828391 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.828393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.828394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.828650 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:46.828850 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.829017 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.829155 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[28481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187046, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[28481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187046, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:10:46.855664 29004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.847001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.27.75.193:36721)
0923 18:10:46.847167 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.847171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.847266 (+ 95us) server_negotiation.cc:407] Connection header received
0923 18:10:46.847428 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.847431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.847481 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.847557 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:46.848249 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.848791 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.849729 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.849958 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.850041 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.852452 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:46.852455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:46.852457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:46.852484 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:46.853920 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.854344 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.854347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.854348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.854446 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.854728 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.854732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.854734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.854912 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:46.855040 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.855382 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.855499 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[28481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[28481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:10:46.882112 29004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:46.872948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.27.75.193:36721)
0923 18:10:46.873183 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:46.873188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:46.873355 (+ 167us) server_negotiation.cc:407] Connection header received
0923 18:10:46.873500 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:46.873503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:46.873546 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:46.873615 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:46.874357 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.874866 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.875778 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:46.875994 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:46.876099 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:46.878789 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:46.878793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:46.878796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:46.878833 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:46.880365 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.880818 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.880821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.880822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.880903 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:46.881180 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:46.881183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:46.881185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:46.881413 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:46.881554 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:46.881847 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 18:10:46.881967 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260923 18:10:46.883473 28632 tablet_service.cc:2908] Rejecting scan request for tablet 0c66ffb225b94f1797e7dc419e503d10: Uninitialized: clean time has not yet been initialized
I20260923 18:10:46.885560 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 28588
I20260923 18:10:46.891113 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 28724
I20260923 18:10:46.896065 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 28858
I20260923 18:10:46.900897 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 28494
2026-09-23T18:10:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2260 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[29031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[29031](info): set up 2 sockets
Sep 23 18:10:46 dist-test-slave-wsx5 krb5kdc[29031](info): commencing operation
krb5kdc: starting...
W20260923 18:10:47.859009 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:10:47 dist-test-slave-wsx5 krb5kdc[29031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:10:47.982456 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:43703
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:48.094517 29044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:48.094815 29044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:48.094856 29044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:48.098979 29044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:48.099037 29044 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 18:10:48.099053 29044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:48.099071 29044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:48.099086 29044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:48.099102 29044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:48.104473 29044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:43703
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:48.105567 29044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:48.106603 29044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:48.115306 29049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:48.115306 29050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:48.115422 29044 server_base.cc:1034] running on GCE node
W20260923 18:10:48.115326 29052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:48.116298 29044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:48.117611 29044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:48.118840 29044 hybrid_clock.cc:648] HybridClock initialized: now 1790187048118814 us; error 33 us; skew 500 ppm
Sep 23 18:10:48 dist-test-slave-wsx5 krb5kdc[29031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:48.126049 29044 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:10:48.127045 29044 webserver.cc:460] Webserver started at http://127.27.75.254:33681/ using document root <none> and password file <none>
I20260923 18:10:48.127611 29044 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:48.127674 29044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:48.127898 29044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:48.130136 29044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "68c77781a41e4297a5d31563cecbaa72"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "00d602a7a4b4fde766023e18ed4a2ca1"
server_key_iv: "9c2022ad59f191b6c0f2f1a15cf25f28"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.130683 29044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "68c77781a41e4297a5d31563cecbaa72"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "00d602a7a4b4fde766023e18ed4a2ca1"
server_key_iv: "9c2022ad59f191b6c0f2f1a15cf25f28"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.134516 29044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:10:48.137101 29059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:48.138550 29044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 18:10:48.138674 29044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "68c77781a41e4297a5d31563cecbaa72"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "00d602a7a4b4fde766023e18ed4a2ca1"
server_key_iv: "9c2022ad59f191b6c0f2f1a15cf25f28"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.138782 29044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:48.153214 29044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:48.156478 29044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:48.156665 29044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:48.164247 29044 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:43703
I20260923 18:10:48.164258 29121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:43703 every 8 connection(s)
I20260923 18:10:48.165220 29044 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:48.168007 29122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:48.169013 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 29044
I20260923 18:10:48.169143 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:10:48.169421 27951 external_mini_cluster.cc:1428] Setting key 2afc288d8e9ed7cd4c281432c760068b
I20260923 18:10:48.173990 29122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72: Bootstrap starting.
I20260923 18:10:48.176218 29122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:48.177042 29122 log.cc:824] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:10:48 dist-test-slave-wsx5 krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:48.178963 29122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72: No bootstrap required, opened a new log
I20260923 18:10:48.182137 29122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c77781a41e4297a5d31563cecbaa72" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43703 } }
I20260923 18:10:48.182391 29122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:48.182430 29122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c77781a41e4297a5d31563cecbaa72, State: Initialized, Role: FOLLOWER
I20260923 18:10:48.182886 29122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c77781a41e4297a5d31563cecbaa72" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43703 } }
I20260923 18:10:48.183024 29122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:48.183130 29122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:48.183238 29122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:48.184092 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.170810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.27.75.254:43703)
0923 18:10:48.171251 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.171271 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.171468 (+ 197us) server_negotiation.cc:407] Connection header received
0923 18:10:48.172212 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.172229 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.172585 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.172907 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:48.173734 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.174741 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.175582 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.176056 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.176198 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.178732 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:48.178737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:48.178751 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:48.178779 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:48.181189 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.181651 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.181656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.181661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.181783 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.182055 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.182058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.182059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.182521 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:48.182719 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.182999 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.183309 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":124,"threads_started":1}
I20260923 18:10:48.184150 29122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c77781a41e4297a5d31563cecbaa72" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43703 } }
I20260923 18:10:48.184496 29122 leader_election.cc:302] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68c77781a41e4297a5d31563cecbaa72; no voters:
I20260923 18:10:48.184795 29122 leader_election.cc:288] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:48.184911 29127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:48.185178 29127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [term 1 LEADER]: Becoming Leader. State: Replica: 68c77781a41e4297a5d31563cecbaa72, State: Running, Role: LEADER
I20260923 18:10:48.185567 29127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c77781a41e4297a5d31563cecbaa72" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43703 } }
I20260923 18:10:48.186107 29122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:48.186784 29129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c77781a41e4297a5d31563cecbaa72. Latest consensus state: current_term: 1 leader_uuid: "68c77781a41e4297a5d31563cecbaa72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c77781a41e4297a5d31563cecbaa72" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43703 } } }
I20260923 18:10:48.186935 29129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:48.187330 29137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:48.187602 29130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68c77781a41e4297a5d31563cecbaa72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c77781a41e4297a5d31563cecbaa72" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43703 } } }
I20260923 18:10:48.187772 29130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:48.191048 29137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:48.196844 29137 catalog_manager.cc:1347] Generated new cluster ID: 355092bd087f4497b7b62e629eb1996f
I20260923 18:10:48.196934 29137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:48.208154 29137 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:48.209489 29137 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:48.219159 29137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72: Generated new TSK 0
I20260923 18:10:48.220065 29137 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:10:48.252259 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43703
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:48.364643 29149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:48.364938 29149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:48.364984 29149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:48.369076 29149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:48.369165 29149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:48.369206 29149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:10:48.369246 29149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:10:48.374682 29149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.75.254:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:48.375857 29149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:48.376907 29149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:48.386674 29154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:48.386766 29155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:48.386979 29157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:48.387163 29149 server_base.cc:1034] running on GCE node
I20260923 18:10:48.387567 29149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:48.388243 29149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:48.389436 29149 hybrid_clock.cc:648] HybridClock initialized: now 1790187048389412 us; error 35 us; skew 500 ppm
Sep 23 18:10:48 dist-test-slave-wsx5 krb5kdc[29031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:48.392536 29149 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:10:48.393584 29149 webserver.cc:460] Webserver started at http://127.27.75.193:39943/ using document root <none> and password file <none>
I20260923 18:10:48.394171 29149 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:48.394237 29149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:48.394443 29149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:48.396623 29149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "0ba7a8e5d7ab48658931ef2c3f498575"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "d87eede725b66ab69695475cd0714973"
server_key_iv: "818a6ada4bd583dc9cf738f9211a92fc"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.397248 29149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "0ba7a8e5d7ab48658931ef2c3f498575"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "d87eede725b66ab69695475cd0714973"
server_key_iv: "818a6ada4bd583dc9cf738f9211a92fc"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.401221 29149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:10:48.403772 29164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:48.405073 29149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:10:48.405217 29149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "0ba7a8e5d7ab48658931ef2c3f498575"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "d87eede725b66ab69695475cd0714973"
server_key_iv: "818a6ada4bd583dc9cf738f9211a92fc"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.405323 29149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:48.434458 29149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:48.437834 29149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:48.438022 29149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:48.439456 29149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:48.439514 29149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:48.439569 29149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:48.439595 29149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:10:48 dist-test-slave-wsx5 krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:48.449571 29149 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:38023
I20260923 18:10:48.449669 29280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:38023 every 8 connection(s)
I20260923 18:10:48.450462 29149 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:10:48.453817 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.440528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:57237 (local address 127.27.75.254:43703)
0923 18:10:48.440649 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.440653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.441771 (+ 1118us) server_negotiation.cc:407] Connection header received
0923 18:10:48.443042 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.443047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.443110 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.443210 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:48.444659 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.445441 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.446623 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.446866 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.446957 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.449886 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:48.449890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:48.449893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:48.449922 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:48.451607 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.452174 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.452177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.452179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.452284 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.452632 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.452634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.452635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.452847 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:48.452929 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.453536 (+ 607us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.453677 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 18:10:48.454676 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.441069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43703 (local address 127.27.75.193:57237)
0923 18:10:48.441609 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:48.441663 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:48.442774 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:48.443371 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:48.443378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:48.443837 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:48.444515 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.444532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.445566 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.445570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.446505 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.446515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.447010 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.447014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.447189 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.447791 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:48.447814 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:48.449708 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:48.451739 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.451745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.451762 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.452046 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.452398 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.452401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.452402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.452546 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.452952 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:48.452958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:48.453283 (+ 325us) client_negotiation.cc:769] Sending connection context
0923 18:10:48.453454 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 18:10:48.453744 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":124,"threads_started":1}
I20260923 18:10:48.456223 29281 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43703
I20260923 18:10:48.456389 29281 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:48.457080 29281 heartbeater.cc:511] Master 127.27.75.254:43703 requested a full tablet report, sending...
I20260923 18:10:48.459008 29075 ts_manager.cc:194] Registered new tserver with Master: 0ba7a8e5d7ab48658931ef2c3f498575 (127.27.75.193:38023)
I20260923 18:10:48.459282 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 29149
I20260923 18:10:48.459380 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:10:48.459666 27951 external_mini_cluster.cc:1428] Setting key f254c7cd0f9c409cbcbf6d76fa5b6359
I20260923 18:10:48.460530 29075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:57237
I20260923 18:10:48.466775 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.458602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:59197 (local address 127.27.75.254:43703)
0923 18:10:48.458727 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.458731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.458822 (+ 91us) server_negotiation.cc:407] Connection header received
0923 18:10:48.458919 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.458922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.458969 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.459081 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:48.459605 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.460425 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.461422 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.461718 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.461833 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.463186 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:48.463189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:48.463191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:48.463221 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:48.465086 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.465636 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.465639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.465640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.465732 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.466012 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.466014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.466015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.466193 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:48.466263 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.466549 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.466649 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":52}
I20260923 18:10:48.466879 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.458482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43703 (local address 127.27.75.193:59197)
0923 18:10:48.458744 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:48.458759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:48.458845 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:48.459096 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:48.459099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:48.459356 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:48.459484 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.459492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.460544 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.460548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.461324 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.461333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.461853 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.461856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.461963 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.462624 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:48.462638 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:48.463042 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:48.465235 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.465239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.465242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.465517 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.465826 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.465829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.465831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.465931 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.466276 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:48.466279 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:48.466389 (+ 110us) client_negotiation.cc:769] Sending connection context
0923 18:10:48.466510 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 18:10:48.466669 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":76,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:10:48.488075 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43703
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:48.599421 29290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:48.599711 29290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:48.599756 29290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:48.603861 29290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:48.603936 29290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:48.603978 29290 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:10:48.604012 29290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:10:48.609481 29290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.75.254:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:48.610603 29290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:48.611629 29290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:48.621359 29295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:48.621387 29296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:48.621390 29298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:48.621371 29290 server_base.cc:1034] running on GCE node
I20260923 18:10:48.622013 29290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:48.622689 29290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:48.623865 29290 hybrid_clock.cc:648] HybridClock initialized: now 1790187048623822 us; error 44 us; skew 500 ppm
Sep 23 18:10:48 dist-test-slave-wsx5 krb5kdc[29031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:48.626760 29290 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:10:48.627676 29290 webserver.cc:460] Webserver started at http://127.27.75.194:38393/ using document root <none> and password file <none>
I20260923 18:10:48.628211 29290 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:48.628306 29290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:48.628520 29290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:48.630722 29290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "f848ff76c9b6411fa7837c8d6380ebdb"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "c4dd1b8f95e2bca4e89533acb344f808"
server_key_iv: "79aa41fdb30e9226c22984166282a9bc"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.631307 29290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "f848ff76c9b6411fa7837c8d6380ebdb"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "c4dd1b8f95e2bca4e89533acb344f808"
server_key_iv: "79aa41fdb30e9226c22984166282a9bc"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.635073 29290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:10:48.637638 29305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:48.638906 29290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:10:48.639017 29290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "f848ff76c9b6411fa7837c8d6380ebdb"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "c4dd1b8f95e2bca4e89533acb344f808"
server_key_iv: "79aa41fdb30e9226c22984166282a9bc"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.639115 29290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:48.663872 29290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:48.667166 29290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:48.667361 29290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:48.668859 29290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:48.668908 29290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:48.668962 29290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:48.668989 29290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:10:48 dist-test-slave-wsx5 krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:48.678884 29290 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:46071
I20260923 18:10:48.678915 29421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:46071 every 8 connection(s)
I20260923 18:10:48.679808 29290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:10:48.682606 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.669664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:47285 (local address 127.27.75.254:43703)
0923 18:10:48.669866 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.669870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.670746 (+ 876us) server_negotiation.cc:407] Connection header received
0923 18:10:48.671958 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.671961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.672018 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.672114 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:48.673826 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.674641 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.676006 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.676252 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.676346 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.679235 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:48.679239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:48.679242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:48.679272 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:48.680725 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.681239 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.681242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.681243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.681322 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.681603 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.681606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.681607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.681852 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:48.681933 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.682382 (+ 449us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.682487 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260923 18:10:48.683357 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.670066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43703 (local address 127.27.75.194:47285)
0923 18:10:48.670562 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:48.670620 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:48.671710 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:48.672338 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:48.672349 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:48.672821 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:48.673602 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.673623 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.674824 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.674827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.675886 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.675898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.676457 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.676463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.676624 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.677257 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:48.677279 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:48.679043 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:48.680876 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.680880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.680891 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.681154 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.681386 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.681388 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.681389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.681513 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.681935 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:48.681940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:48.682163 (+ 223us) client_negotiation.cc:769] Sending connection context
0923 18:10:48.682311 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 18:10:48.682554 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260923 18:10:48.684712 29422 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43703
I20260923 18:10:48.684891 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 29290
I20260923 18:10:48.684916 29422 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:48.684994 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:10:48.685299 27951 external_mini_cluster.cc:1428] Setting key eef731a5bfc8968ec2bf1986996ed222
I20260923 18:10:48.685508 29422 heartbeater.cc:511] Master 127.27.75.254:43703 requested a full tablet report, sending...
I20260923 18:10:48.686540 29075 ts_manager.cc:194] Registered new tserver with Master: f848ff76c9b6411fa7837c8d6380ebdb (127.27.75.194:46071)
I20260923 18:10:48.687242 29075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:47285
I20260923 18:10:48.694703 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.687023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:55329 (local address 127.27.75.254:43703)
0923 18:10:48.687128 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.687132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.687710 (+ 578us) server_negotiation.cc:407] Connection header received
0923 18:10:48.687819 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.687822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.687866 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.687945 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:48.688499 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.689039 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.690328 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.690550 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.690631 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.691830 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:48.691833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:48.691835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:48.691857 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:48.693100 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.693526 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.693529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.693531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.693608 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.693883 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.693886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.693888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.694105 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:48.694195 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.694441 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.694545 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 18:10:48.694713 29428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.687154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43703 (local address 127.27.75.194:55329)
0923 18:10:48.687626 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:48.687640 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:48.687738 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:48.687960 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:48.687963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:48.688225 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:48.688362 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.688370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.689225 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.689228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.690170 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.690181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.690667 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.690671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.690777 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.691270 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:48.691284 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:48.691703 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:48.693236 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.693239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.693241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.693438 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.693692 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.693695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.693697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.693778 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.694198 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:48.694202 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:48.694297 (+ 95us) client_negotiation.cc:769] Sending connection context
0923 18:10:48.694381 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 18:10:48.694532 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:10:48.714495 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43703
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:48.827432 29431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:48.827718 29431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:48.827781 29431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:48.832190 29431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:48.832270 29431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:48.832332 29431 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:10:48.832389 29431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:10:48.837980 29431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.75.254:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:48.839074 29431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:48.840093 29431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:48.849874 29436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:48.849866 29439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:48.849918 29437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:48.849885 29431 server_base.cc:1034] running on GCE node
I20260923 18:10:48.850587 29431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:48.851246 29431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:48.852424 29431 hybrid_clock.cc:648] HybridClock initialized: now 1790187048852403 us; error 34 us; skew 500 ppm
Sep 23 18:10:48 dist-test-slave-wsx5 krb5kdc[29031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:48.855111 29431 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:10:48.856046 29431 webserver.cc:460] Webserver started at http://127.27.75.195:40975/ using document root <none> and password file <none>
I20260923 18:10:48.856630 29431 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:48.856699 29431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:48.856910 29431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:48.859061 29431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "5888b362069e4c3d83454faf5faf7f7e"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "852b4ffda26ee6d03813f824f89ed219"
server_key_iv: "dcccce34ab54f707111895219f7d2da2"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.859669 29431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "5888b362069e4c3d83454faf5faf7f7e"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "852b4ffda26ee6d03813f824f89ed219"
server_key_iv: "dcccce34ab54f707111895219f7d2da2"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.863571 29431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 18:10:48.866266 29446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:48.867620 29431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:48.867802 29431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "5888b362069e4c3d83454faf5faf7f7e"
format_stamp: "Formatted at 2026-09-23 18:10:48 on dist-test-slave-wsx5"
server_key: "852b4ffda26ee6d03813f824f89ed219"
server_key_iv: "dcccce34ab54f707111895219f7d2da2"
server_key_version: "encryptionkey@0"
I20260923 18:10:48.867915 29431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:48.883077 29431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:48.886775 29431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:48.886976 29431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:48.888499 29431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:48.888552 29431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:48.888593 29431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:48.888608 29431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:48.898936 29431 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:46001
I20260923 18:10:48.899044 29562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:46001 every 8 connection(s)
Sep 23 18:10:48 dist-test-slave-wsx5 krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:48.900060 29431 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:10:48.900983 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 29431
I20260923 18:10:48.901111 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:10:48.901738 27951 external_mini_cluster.cc:1428] Setting key af0165d78844ccfa1239d20ed2b4f833
I20260923 18:10:48.905007 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.889755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:35283 (local address 127.27.75.254:43703)
0923 18:10:48.889888 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.889891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.890949 (+ 1058us) server_negotiation.cc:407] Connection header received
0923 18:10:48.892129 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.892133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.892193 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.892279 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:48.894125 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.894957 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.896824 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.897210 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.897340 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.900883 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:48.900887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:48.900889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:48.900920 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:48.902415 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.903054 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.903058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.903060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.903175 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.903593 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.903595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.903597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.903922 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:48.904130 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.904713 (+ 583us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.904847 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:10:48.905939 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.890226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43703 (local address 127.27.75.195:35283)
0923 18:10:48.890769 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:48.890833 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:48.891897 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:48.892526 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:48.892537 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:48.893060 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:48.893924 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.893948 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.895114 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.895118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.896631 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.896646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.897554 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.897559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.897794 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.898472 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:48.898492 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:48.900477 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:48.902549 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.902555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.902574 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.902937 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.903318 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.903321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.903323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.903484 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.904120 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:48.904127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:48.904444 (+ 317us) client_negotiation.cc:769] Sending connection context
0923 18:10:48.904656 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 18:10:48.905015 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":118,"threads_started":1}
I20260923 18:10:48.907063 29563 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43703
I20260923 18:10:48.907207 29563 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:48.907797 29563 heartbeater.cc:511] Master 127.27.75.254:43703 requested a full tablet report, sending...
I20260923 18:10:48.909008 29076 ts_manager.cc:194] Registered new tserver with Master: 5888b362069e4c3d83454faf5faf7f7e (127.27.75.195:46001)
I20260923 18:10:48.909641 29076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:35283
I20260923 18:10:48.916411 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:10:48.916810 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.910069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43703 (local address 127.27.75.195:48723)
0923 18:10:48.910342 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:48.910354 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:48.910463 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:48.910666 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:48.910668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:48.910870 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:48.910968 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.910973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.911706 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.911709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.912263 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.912271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.912682 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.912685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.912777 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.913267 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:48.913278 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:48.913611 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:48.915212 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.915216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.915219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.915475 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.915795 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:48.915799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:48.915800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:48.915905 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:48.916301 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:48.916304 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:48.916399 (+ 95us) client_negotiation.cc:769] Sending connection context
0923 18:10:48.916487 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 18:10:48.916623 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":89,"threads_started":1}
I20260923 18:10:48.916829 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.910139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:48723 (local address 127.27.75.254:43703)
0923 18:10:48.910247 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.910252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.910423 (+ 171us) server_negotiation.cc:407] Connection header received
0923 18:10:48.910537 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.910540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.910581 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.910642 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:48.911072 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.911590 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.912374 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.912586 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.912669 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.913747 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:48.913751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:48.913754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:48.913778 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:48.915103 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.915565 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.915569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.915570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.915673 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.915995 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.915998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.916000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.916208 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:48.916301 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.916561 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.916676 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 23 18:10:48 dist-test-slave-wsx5 krb5kdc[29031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 18:10:48 dist-test-slave-wsx5 krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:48.942292 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.933918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.27.75.254:43703)
0923 18:10:48.934077 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.934080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.934092 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:10:48.934134 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.934136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.934175 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.934240 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:48.935052 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.935606 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.936465 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.936679 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.936760 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.939353 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:48.939356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:48.939358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:48.939389 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:48.940530 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.940961 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.940964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.940966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.941050 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:48.941437 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:48.941440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:48.941441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:48.941647 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:48.941756 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.941996 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.942126 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 18:10:48.945415 29076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:10:48.948392 29076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:48.963567 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.957586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.27.75.193:38023)
0923 18:10:48.959057 (+ 1471us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.959063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.959087 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:10:48.959170 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.959174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.959360 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.959504 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:48.959705 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.960477 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.962010 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.962777 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.962897 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.963077 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.963293 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.963411 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1366,"thread_start_us":127,"threads_started":1}
I20260923 18:10:48.963553 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.957440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:38023 (local address 127.0.0.1:40048)
0923 18:10:48.958321 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:48.958389 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:48.958505 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:48.959491 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:48.959494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:48.959511 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:48.959605 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.959610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.960593 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.960596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.961872 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.961884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.962930 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.962934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.963056 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.963108 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 18:10:48.963241 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 18:10:48.963386 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":70,"threads_started":1}
I20260923 18:10:48.963897 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.957739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:46071 (local address 127.0.0.1:47848)
0923 18:10:48.958633 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:48.958648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:48.958757 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:48.959274 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:48.959280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:48.959302 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:48.959446 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.959455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.960930 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.960933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.962217 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.962228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.963490 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.963493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.963587 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.963610 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:10:48.963679 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 18:10:48.963769 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260923 18:10:48.964202 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.958618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44188 (local address 127.27.75.195:46001)
0923 18:10:48.959057 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.959067 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.959087 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:10:48.959158 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.959163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.959360 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.959504 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:48.959902 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.960986 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.962391 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.962987 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.963090 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.963249 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.963625 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.964061 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":128,"threads_started":1}
I20260923 18:10:48.964300 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.957890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.27.75.194:46071)
0923 18:10:48.958295 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.958306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.958858 (+ 552us) server_negotiation.cc:407] Connection header received
0923 18:10:48.958934 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.958938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.959112 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.959254 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:48.959575 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.960763 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.962359 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.963183 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.963286 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.963427 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.964071 (+ 644us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.964184 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":118,"threads_started":1}
I20260923 18:10:48.964532 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.958203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:46001 (local address 127.0.0.1:44188)
0923 18:10:48.958743 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:48.958756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:48.958835 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:48.959654 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:48.959657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:48.959675 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:48.959773 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.959779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.961103 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.961106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.962268 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.962275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.963309 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.963312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.963392 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.963412 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 18:10:48.963476 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 18:10:48.963535 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":83,"threads_started":1}
I20260923 18:10:48.967044 29354 tablet_service.cc:1467] Processing CreateTablet for tablet c05ea645c8f2485b93f90b8ded36382a (DEFAULT_TABLE table=test-table [id=96c9b174206143798fe9f55fbf67c751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:48.967063 29497 tablet_service.cc:1467] Processing CreateTablet for tablet c05ea645c8f2485b93f90b8ded36382a (DEFAULT_TABLE table=test-table [id=96c9b174206143798fe9f55fbf67c751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:48.967062 29214 tablet_service.cc:1467] Processing CreateTablet for tablet c05ea645c8f2485b93f90b8ded36382a (DEFAULT_TABLE table=test-table [id=96c9b174206143798fe9f55fbf67c751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:48.968129 29354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05ea645c8f2485b93f90b8ded36382a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:48.968151 29497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05ea645c8f2485b93f90b8ded36382a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:48.968143 29214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05ea645c8f2485b93f90b8ded36382a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:48.975607 29586 tablet_bootstrap.cc:492] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb: Bootstrap starting.
I20260923 18:10:48.976495 29587 tablet_bootstrap.cc:492] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575: Bootstrap starting.
I20260923 18:10:48.977152 29588 tablet_bootstrap.cc:492] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e: Bootstrap starting.
I20260923 18:10:48.978158 29586 tablet_bootstrap.cc:654] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:48.978722 29587 tablet_bootstrap.cc:654] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:48.978972 29588 tablet_bootstrap.cc:654] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:48.979067 29586 log.cc:824] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:48.979521 29587 log.cc:824] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:48.979744 29588 log.cc:824] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:48.981709 29586 tablet_bootstrap.cc:492] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb: No bootstrap required, opened a new log
I20260923 18:10:48.981794 29588 tablet_bootstrap.cc:492] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e: No bootstrap required, opened a new log
I20260923 18:10:48.981794 29587 tablet_bootstrap.cc:492] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575: No bootstrap required, opened a new log
I20260923 18:10:48.981920 29586 ts_tablet_manager.cc:1397] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 18:10:48.981992 29587 ts_tablet_manager.cc:1397] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:10:48.981985 29588 ts_tablet_manager.cc:1397] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 18:10:48.985222 29586 raft_consensus.cc:348] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:48.985494 29586 raft_consensus.cc:374] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:48.985545 29586 raft_consensus.cc:729] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f848ff76c9b6411fa7837c8d6380ebdb, State: Initialized, Role: FOLLOWER
I20260923 18:10:48.986106 29586 consensus_queue.cc:260] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:48.986083 29588 raft_consensus.cc:348] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:48.986083 29587 raft_consensus.cc:348] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:48.986362 29587 raft_consensus.cc:374] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:48.986362 29588 raft_consensus.cc:374] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:48.986423 29587 raft_consensus.cc:729] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba7a8e5d7ab48658931ef2c3f498575, State: Initialized, Role: FOLLOWER
I20260923 18:10:48.986430 29588 raft_consensus.cc:729] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5888b362069e4c3d83454faf5faf7f7e, State: Initialized, Role: FOLLOWER
I20260923 18:10:48.986910 29588 consensus_queue.cc:260] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:48.986909 29587 consensus_queue.cc:260] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:48.987957 29587 ts_tablet_manager.cc:1428] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:10:48.987957 29588 ts_tablet_manager.cc:1428] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:10:48.987974 29586 ts_tablet_manager.cc:1428] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:10:48.988339 29281 heartbeater.cc:503] Master 127.27.75.254:43703 was elected leader, sending a full tablet report...
I20260923 18:10:48.988341 29422 heartbeater.cc:503] Master 127.27.75.254:43703 was elected leader, sending a full tablet report...
I20260923 18:10:48.989176 29592 raft_consensus.cc:482] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:48.989375 29563 heartbeater.cc:503] Master 127.27.75.254:43703 was elected leader, sending a full tablet report...
I20260923 18:10:48.989364 29592 raft_consensus.cc:504] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:48.990626 29592 leader_election.cc:288] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ba7a8e5d7ab48658931ef2c3f498575 (127.27.75.193:38023), 5888b362069e4c3d83454faf5faf7f7e (127.27.75.195:46001)
I20260923 18:10:48.994971 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.990826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:38023 (local address 127.27.75.194:42645)
0923 18:10:48.991071 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:48.991087 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:48.991172 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:48.991682 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:48.991685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:48.991706 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:48.991819 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.991824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.992663 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.992666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.993820 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.993830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.994556 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.994560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.994667 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.994685 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 18:10:48.994762 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 18:10:48.994836 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260923 18:10:48.994971 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.991329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:42645 (local address 127.27.75.193:38023)
0923 18:10:48.991470 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.991474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.991489 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:10:48.991531 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.991533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.991567 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.991646 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:48.991925 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.992538 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.993940 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.994446 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.994556 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.994607 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.994788 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.994857 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 18:10:48.995651 29235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05ea645c8f2485b93f90b8ded36382a" candidate_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" is_pre_election: true
I20260923 18:10:48.996070 29235 raft_consensus.cc:2454] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f848ff76c9b6411fa7837c8d6380ebdb in term 0.
I20260923 18:10:48.996575 29307 leader_election.cc:302] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba7a8e5d7ab48658931ef2c3f498575, f848ff76c9b6411fa7837c8d6380ebdb; no voters:
I20260923 18:10:48.996819 29592 raft_consensus.cc:2781] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:48.996918 29592 raft_consensus.cc:482] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:48.996961 29592 raft_consensus.cc:3037] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:48.997865 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.990884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:46001 (local address 127.27.75.194:40015)
0923 18:10:48.992650 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:48.992664 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:48.992829 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:48.993195 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:48.993197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:48.993213 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:48.994459 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.994468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.995568 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.995572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.996589 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:48.996598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.997484 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.997487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:48.997564 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.997586 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:10:48.997649 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 18:10:48.997750 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1700,"thread_start_us":111,"threads_started":1}
I20260923 18:10:48.997869 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:48.991015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:40015 (local address 127.27.75.195:46001)
0923 18:10:48.991132 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:48.991137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:48.992760 (+ 1623us) server_negotiation.cc:407] Connection header received
0923 18:10:48.992942 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:48.992945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:48.992990 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:48.993032 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:48.994569 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.995154 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.996756 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:48.997332 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:48.997444 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:48.997492 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:48.997670 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 18:10:48.997738 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:10:48.998215 29592 raft_consensus.cc:504] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:48.998492 29517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05ea645c8f2485b93f90b8ded36382a" candidate_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5888b362069e4c3d83454faf5faf7f7e" is_pre_election: true
I20260923 18:10:48.998550 29592 leader_election.cc:288] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [CANDIDATE]: Term 1 election: Requested vote from peers 0ba7a8e5d7ab48658931ef2c3f498575 (127.27.75.193:38023), 5888b362069e4c3d83454faf5faf7f7e (127.27.75.195:46001)
I20260923 18:10:48.998786 29517 raft_consensus.cc:2454] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f848ff76c9b6411fa7837c8d6380ebdb in term 0.
I20260923 18:10:48.998867 29235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05ea645c8f2485b93f90b8ded36382a" candidate_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba7a8e5d7ab48658931ef2c3f498575"
I20260923 18:10:48.998885 29516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c05ea645c8f2485b93f90b8ded36382a" candidate_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5888b362069e4c3d83454faf5faf7f7e"
I20260923 18:10:48.998989 29516 raft_consensus.cc:3037] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:48.998991 29235 raft_consensus.cc:3037] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:49.000253 29235 raft_consensus.cc:2454] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f848ff76c9b6411fa7837c8d6380ebdb in term 1.
I20260923 18:10:49.000458 29516 raft_consensus.cc:2454] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f848ff76c9b6411fa7837c8d6380ebdb in term 1.
I20260923 18:10:49.000568 29307 leader_election.cc:302] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba7a8e5d7ab48658931ef2c3f498575, f848ff76c9b6411fa7837c8d6380ebdb; no voters:
I20260923 18:10:49.000744 29592 raft_consensus.cc:2781] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:49.000953 29592 raft_consensus.cc:686] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 1 LEADER]: Becoming Leader. State: Replica: f848ff76c9b6411fa7837c8d6380ebdb, State: Running, Role: LEADER
I20260923 18:10:49.001305 29592 consensus_queue.cc:237] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.004362 29075 catalog_manager.cc:5495] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb reported cstate change: term changed from 0 to 1, leader changed from <none> to f848ff76c9b6411fa7837c8d6380ebdb (127.27.75.194). New cstate: current_term: 1 leader_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } health_report { overall_health: UNKNOWN } } }
I20260923 18:10:49.034246 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.029992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.27.75.194:46071)
0923 18:10:49.030121 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:49.030126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:49.030303 (+ 177us) server_negotiation.cc:407] Connection header received
0923 18:10:49.030483 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:49.030485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:49.030545 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:49.030630 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:49.030990 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.031541 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.032558 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.032896 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.033006 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.033359 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:49.033860 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:49.033951 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:49.034048 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 18:10:49.034101 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 18:10:49.040499 29235 raft_consensus.cc:1261] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575 [term 1 FOLLOWER]: Refusing update from remote peer f848ff76c9b6411fa7837c8d6380ebdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:49.040499 29516 raft_consensus.cc:1261] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e [term 1 FOLLOWER]: Refusing update from remote peer f848ff76c9b6411fa7837c8d6380ebdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:49.041249 29596 consensus_queue.cc:1035] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:49.041451 29592 consensus_queue.cc:1035] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:49.048836 29599 mvcc.cc:205] Tried to move back new op lower bound from 7332606152862023680 to 7332606152711888896. Current Snapshot: MvccSnapshot[applied={T|T < 7332606152862023680}]
I20260923 18:10:49.061302 29075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:49.061476 29075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:49.064082 29075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72: Sending DeleteTablet for 3 replicas of tablet c05ea645c8f2485b93f90b8ded36382a
I20260923 18:10:49.065107 29497 tablet_service.cc:1514] Processing DeleteTablet for tablet c05ea645c8f2485b93f90b8ded36382a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:49 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:44188
I20260923 18:10:49.065093 29354 tablet_service.cc:1514] Processing DeleteTablet for tablet c05ea645c8f2485b93f90b8ded36382a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:49 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:47848
I20260923 18:10:49.065677 29214 tablet_service.cc:1514] Processing DeleteTablet for tablet c05ea645c8f2485b93f90b8ded36382a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:49 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:40048
I20260923 18:10:49.065956 29608 tablet_replica.cc:330] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e: stopping tablet replica
I20260923 18:10:49.066181 29610 tablet_replica.cc:330] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575: stopping tablet replica
I20260923 18:10:49.066336 29608 raft_consensus.cc:2229] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:49.066262 29075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 18:10:49.066473 29610 raft_consensus.cc:2229] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:49.066684 29608 raft_consensus.cc:2258] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 18:10:49.066728 29075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:49.066744 29610 raft_consensus.cc:2258] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:49.067071 29609 tablet_replica.cc:330] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb: stopping tablet replica
I20260923 18:10:49.067356 29609 raft_consensus.cc:2229] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:10:49.067749 29609 raft_consensus.cc:2258] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:49.068879 29610 ts_tablet_manager.cc:1905] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:49.069152 29609 ts_tablet_manager.cc:1905] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:49.070148 29608 ts_tablet_manager.cc:1905] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:49.071875 29609 ts_tablet_manager.cc:1918] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:10:49.071967 29609 log.cc:1196] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/c05ea645c8f2485b93f90b8ded36382a
I20260923 18:10:49.072365 29609 ts_tablet_manager.cc:1939] T c05ea645c8f2485b93f90b8ded36382a P f848ff76c9b6411fa7837c8d6380ebdb: Deleting consensus metadata
I20260923 18:10:49.072402 29214 tablet_service.cc:1467] Processing CreateTablet for tablet caf8414807904c3ea90eac648798caa7 (DEFAULT_TABLE table=test-table [id=ccd36ba01fa54342b207f399bc21db5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:49.072647 29214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf8414807904c3ea90eac648798caa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:49.073891 29062 catalog_manager.cc:4841] TS f848ff76c9b6411fa7837c8d6380ebdb (127.27.75.194:46071): tablet c05ea645c8f2485b93f90b8ded36382a (table test-table [id=96c9b174206143798fe9f55fbf67c751]) successfully deleted
I20260923 18:10:49.074563 29354 tablet_service.cc:1467] Processing CreateTablet for tablet caf8414807904c3ea90eac648798caa7 (DEFAULT_TABLE table=test-table [id=ccd36ba01fa54342b207f399bc21db5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:49.074805 29354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf8414807904c3ea90eac648798caa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:49.074977 29610 ts_tablet_manager.cc:1918] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:10:49.075079 29610 log.cc:1196] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/wal/wals/c05ea645c8f2485b93f90b8ded36382a
I20260923 18:10:49.075337 29610 ts_tablet_manager.cc:1939] T c05ea645c8f2485b93f90b8ded36382a P 0ba7a8e5d7ab48658931ef2c3f498575: Deleting consensus metadata
I20260923 18:10:49.076387 29061 catalog_manager.cc:4841] TS 0ba7a8e5d7ab48658931ef2c3f498575 (127.27.75.193:38023): tablet c05ea645c8f2485b93f90b8ded36382a (table test-table [id=96c9b174206143798fe9f55fbf67c751]) successfully deleted
I20260923 18:10:49.076953 29586 tablet_bootstrap.cc:492] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb: Bootstrap starting.
I20260923 18:10:49.077004 29587 tablet_bootstrap.cc:492] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575: Bootstrap starting.
I20260923 18:10:49.077142 29608 ts_tablet_manager.cc:1918] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:10:49.077237 29608 log.cc:1196] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/c05ea645c8f2485b93f90b8ded36382a
I20260923 18:10:49.077523 29608 ts_tablet_manager.cc:1939] T c05ea645c8f2485b93f90b8ded36382a P 5888b362069e4c3d83454faf5faf7f7e: Deleting consensus metadata
I20260923 18:10:49.078009 29587 tablet_bootstrap.cc:654] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:49.078050 29586 tablet_bootstrap.cc:654] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:49.078902 29497 tablet_service.cc:1467] Processing CreateTablet for tablet caf8414807904c3ea90eac648798caa7 (DEFAULT_TABLE table=test-table [id=ccd36ba01fa54342b207f399bc21db5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:49.079140 29497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf8414807904c3ea90eac648798caa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:49.079298 29587 tablet_bootstrap.cc:492] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575: No bootstrap required, opened a new log
I20260923 18:10:49.079378 29587 ts_tablet_manager.cc:1397] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:10:49.079622 29586 tablet_bootstrap.cc:492] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb: No bootstrap required, opened a new log
I20260923 18:10:49.079686 29586 ts_tablet_manager.cc:1397] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:10:49.079946 29587 raft_consensus.cc:348] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.080051 29587 raft_consensus.cc:374] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:49.080083 29587 raft_consensus.cc:729] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba7a8e5d7ab48658931ef2c3f498575, State: Initialized, Role: FOLLOWER
I20260923 18:10:49.080077 29586 raft_consensus.cc:348] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.080165 29586 raft_consensus.cc:374] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:49.080193 29586 raft_consensus.cc:729] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f848ff76c9b6411fa7837c8d6380ebdb, State: Initialized, Role: FOLLOWER
I20260923 18:10:49.080201 29587 consensus_queue.cc:260] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.080303 29586 consensus_queue.cc:260] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.080549 29586 ts_tablet_manager.cc:1428] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:10:49.081245 29588 tablet_bootstrap.cc:492] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e: Bootstrap starting.
I20260923 18:10:49.081416 29062 catalog_manager.cc:4841] TS 5888b362069e4c3d83454faf5faf7f7e (127.27.75.195:46001): tablet c05ea645c8f2485b93f90b8ded36382a (table test-table [id=96c9b174206143798fe9f55fbf67c751]) successfully deleted
I20260923 18:10:49.081769 29587 ts_tablet_manager.cc:1428] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:10:49.082196 29588 tablet_bootstrap.cc:654] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:49.083412 29588 tablet_bootstrap.cc:492] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e: No bootstrap required, opened a new log
I20260923 18:10:49.083494 29588 ts_tablet_manager.cc:1397] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:10:49.083917 29588 raft_consensus.cc:348] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.084035 29588 raft_consensus.cc:374] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:49.084066 29588 raft_consensus.cc:729] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5888b362069e4c3d83454faf5faf7f7e, State: Initialized, Role: FOLLOWER
I20260923 18:10:49.084188 29588 consensus_queue.cc:260] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.084429 29588 ts_tablet_manager.cc:1428] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260923 18:10:49.151163 29564 tablet.cc:2369] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:10:49.181557 29423 tablet.cc:2369] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:10:49.201807 29282 tablet.cc:2369] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:10:49.223737 29593 raft_consensus.cc:482] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:49.223889 29593 raft_consensus.cc:504] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.225050 29593 leader_election.cc:288] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ba7a8e5d7ab48658931ef2c3f498575 (127.27.75.193:38023), f848ff76c9b6411fa7837c8d6380ebdb (127.27.75.194:46071)
I20260923 18:10:49.228628 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.225459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:41977 (local address 127.27.75.193:38023)
0923 18:10:49.225589 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:49.225594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:49.225670 (+ 76us) server_negotiation.cc:407] Connection header received
0923 18:10:49.225810 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:49.225813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:49.225852 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:49.225912 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:49.226150 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.226646 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.227694 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.228082 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.228189 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.228253 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:49.228419 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 18:10:49.228489 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:10:49.228626 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.225355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:38023 (local address 127.27.75.195:41977)
0923 18:10:49.225572 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:49.225587 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:49.225710 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:49.225931 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:49.225934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:49.225949 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:49.226057 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.226062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.226759 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.226763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.227574 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.227583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.228189 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.228192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.228299 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.228326 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 18:10:49.228396 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 18:10:49.228489 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":3}
I20260923 18:10:49.228812 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.225444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:42723 (local address 127.27.75.194:46071)
0923 18:10:49.225625 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:49.225629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:49.225840 (+ 211us) server_negotiation.cc:407] Connection header received
0923 18:10:49.225957 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:49.225960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:49.226003 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:49.226063 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:49.226320 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.226833 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.227874 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.228269 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.228374 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.228428 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:49.228605 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 18:10:49.228678 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 18:10:49.228966 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.225354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:46071 (local address 127.27.75.195:42723)
0923 18:10:49.225739 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:49.225749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:49.225859 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:49.226085 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:49.226088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:49.226104 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:49.226218 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.226223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.226941 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.226945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.227758 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.227767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.228389 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.228392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.228492 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.228513 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:10:49.228577 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 18:10:49.228644 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":65,"thread_start_us":71,"threads_started":1}
I20260923 18:10:49.229318 29235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf8414807904c3ea90eac648798caa7" candidate_uuid: "5888b362069e4c3d83454faf5faf7f7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" is_pre_election: true
I20260923 18:10:49.229477 29235 raft_consensus.cc:2454] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5888b362069e4c3d83454faf5faf7f7e in term 0.
I20260923 18:10:49.229477 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf8414807904c3ea90eac648798caa7" candidate_uuid: "5888b362069e4c3d83454faf5faf7f7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" is_pre_election: true
I20260923 18:10:49.229719 29375 raft_consensus.cc:2454] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5888b362069e4c3d83454faf5faf7f7e in term 0.
I20260923 18:10:49.229925 29448 leader_election.cc:302] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba7a8e5d7ab48658931ef2c3f498575, 5888b362069e4c3d83454faf5faf7f7e; no voters:
I20260923 18:10:49.230206 29593 raft_consensus.cc:2781] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:49.230273 29593 raft_consensus.cc:482] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:49.230376 29593 raft_consensus.cc:3037] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:49.231442 29593 raft_consensus.cc:504] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.231770 29593 leader_election.cc:288] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [CANDIDATE]: Term 1 election: Requested vote from peers 0ba7a8e5d7ab48658931ef2c3f498575 (127.27.75.193:38023), f848ff76c9b6411fa7837c8d6380ebdb (127.27.75.194:46071)
I20260923 18:10:49.232155 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf8414807904c3ea90eac648798caa7" candidate_uuid: "5888b362069e4c3d83454faf5faf7f7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f848ff76c9b6411fa7837c8d6380ebdb"
I20260923 18:10:49.232270 29235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf8414807904c3ea90eac648798caa7" candidate_uuid: "5888b362069e4c3d83454faf5faf7f7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba7a8e5d7ab48658931ef2c3f498575"
I20260923 18:10:49.232338 29375 raft_consensus.cc:3037] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:49.232384 29235 raft_consensus.cc:3037] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:49.233553 29375 raft_consensus.cc:2454] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5888b362069e4c3d83454faf5faf7f7e in term 1.
I20260923 18:10:49.233553 29235 raft_consensus.cc:2454] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5888b362069e4c3d83454faf5faf7f7e in term 1.
I20260923 18:10:49.233860 29449 leader_election.cc:302] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ba7a8e5d7ab48658931ef2c3f498575, 5888b362069e4c3d83454faf5faf7f7e, f848ff76c9b6411fa7837c8d6380ebdb; no voters:
I20260923 18:10:49.234030 29593 raft_consensus.cc:2781] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:49.234230 29593 raft_consensus.cc:686] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 1 LEADER]: Becoming Leader. State: Replica: 5888b362069e4c3d83454faf5faf7f7e, State: Running, Role: LEADER
I20260923 18:10:49.234479 29593 consensus_queue.cc:237] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.236935 29076 catalog_manager.cc:5495] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5888b362069e4c3d83454faf5faf7f7e (127.27.75.195). New cstate: current_term: 1 leader_uuid: "5888b362069e4c3d83454faf5faf7f7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } health_report { overall_health: HEALTHY } } }
Sep 23 18:10:49 dist-test-slave-wsx5 krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:49.299552 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.290913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43703 (local address 127.0.0.1:53464)
0923 18:10:49.291286 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:49.291299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:49.291424 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:49.291644 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:49.291647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:49.292068 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:49.292169 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.292176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.292918 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.292921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.293517 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.293526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.294010 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.294013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.294084 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.294643 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:49.294665 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:49.296262 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:49.297921 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:49.297932 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:49.297938 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:49.298192 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:49.298477 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:49.298480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:49.298481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:49.298567 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:49.299014 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:49.299019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:49.299133 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 18:10:49.299220 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 18:10:49.299362 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260923 18:10:49.299559 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.291031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.27.75.254:43703)
0923 18:10:49.291201 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:49.291206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:49.291376 (+ 170us) server_negotiation.cc:407] Connection header received
0923 18:10:49.291510 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:49.291513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:49.291551 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:49.291615 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:49.292296 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.292815 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.293629 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.293897 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.293994 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.296410 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:49.296414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:49.296415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:49.296440 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:49.297800 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:49.298301 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:49.298304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:49.298306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:49.298383 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:49.298671 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:49.298674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:49.298676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:49.298926 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:49.299009 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:49.299297 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 18:10:49.299427 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 18:10:49.303802 29076 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:53464:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 18:10:49.304502 29076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:49.309762 29354 tablet_service.cc:1467] Processing CreateTablet for tablet afda025f25f14ff597a9a55a4d8367b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=897c30ce669147738188be46280e6bab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:10:49.310007 29497 tablet_service.cc:1467] Processing CreateTablet for tablet afda025f25f14ff597a9a55a4d8367b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=897c30ce669147738188be46280e6bab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:10:49.310190 29354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afda025f25f14ff597a9a55a4d8367b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:49.310390 29497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afda025f25f14ff597a9a55a4d8367b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:49.312567 29586 tablet_bootstrap.cc:492] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: Bootstrap starting.
I20260923 18:10:49.312654 29588 tablet_bootstrap.cc:492] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e: Bootstrap starting.
I20260923 18:10:49.312920 29214 tablet_service.cc:1467] Processing CreateTablet for tablet afda025f25f14ff597a9a55a4d8367b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=897c30ce669147738188be46280e6bab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:10:49.313313 29214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afda025f25f14ff597a9a55a4d8367b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:49.313493 29586 tablet_bootstrap.cc:654] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:49.313511 29588 tablet_bootstrap.cc:654] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:49.314971 29588 tablet_bootstrap.cc:492] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e: No bootstrap required, opened a new log
I20260923 18:10:49.315068 29588 ts_tablet_manager.cc:1397] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:10:49.315572 29588 raft_consensus.cc:348] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.315701 29588 raft_consensus.cc:374] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:49.315738 29588 raft_consensus.cc:729] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5888b362069e4c3d83454faf5faf7f7e, State: Initialized, Role: FOLLOWER
I20260923 18:10:49.315879 29588 consensus_queue.cc:260] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.316162 29588 ts_tablet_manager.cc:1428] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:10:49.316131 29593 tablet_replica.cc:439] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } } }
I20260923 18:10:49.316267 29593 tablet_replica.cc:442] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:49.316478 29587 tablet_bootstrap.cc:492] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575: Bootstrap starting.
I20260923 18:10:49.317401 29586 tablet_bootstrap.cc:492] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: No bootstrap required, opened a new log
I20260923 18:10:49.317476 29586 ts_tablet_manager.cc:1397] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260923 18:10:49.317553 29587 tablet_bootstrap.cc:654] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:49.317879 29586 raft_consensus.cc:348] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.317992 29586 raft_consensus.cc:374] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:49.318022 29586 raft_consensus.cc:729] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f848ff76c9b6411fa7837c8d6380ebdb, State: Initialized, Role: FOLLOWER
I20260923 18:10:49.318138 29586 consensus_queue.cc:260] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.318427 29586 ts_tablet_manager.cc:1428] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:10:49.318395 29596 tablet_replica.cc:439] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } } }
I20260923 18:10:49.318540 29596 tablet_replica.cc:442] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:49.319180 29587 tablet_bootstrap.cc:492] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575: No bootstrap required, opened a new log
I20260923 18:10:49.319264 29587 ts_tablet_manager.cc:1397] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:10:49.319677 29587 raft_consensus.cc:348] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.319780 29587 raft_consensus.cc:374] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:49.319811 29587 raft_consensus.cc:729] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba7a8e5d7ab48658931ef2c3f498575, State: Initialized, Role: FOLLOWER
I20260923 18:10:49.319940 29587 consensus_queue.cc:260] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.320233 29587 ts_tablet_manager.cc:1428] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:10:49.320189 29594 tablet_replica.cc:439] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } } }
I20260923 18:10:49.320344 29594 tablet_replica.cc:442] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:49.360802 29596 raft_consensus.cc:482] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:49.360951 29596 raft_consensus.cc:504] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.361430 29596 leader_election.cc:288] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ba7a8e5d7ab48658931ef2c3f498575 (127.27.75.193:38023), 5888b362069e4c3d83454faf5faf7f7e (127.27.75.195:46001)
I20260923 18:10:49.361884 29235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afda025f25f14ff597a9a55a4d8367b0" candidate_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" is_pre_election: true
I20260923 18:10:49.361898 29516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afda025f25f14ff597a9a55a4d8367b0" candidate_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5888b362069e4c3d83454faf5faf7f7e" is_pre_election: true
I20260923 18:10:49.362025 29516 raft_consensus.cc:2454] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f848ff76c9b6411fa7837c8d6380ebdb in term 0.
I20260923 18:10:49.362036 29235 raft_consensus.cc:2454] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f848ff76c9b6411fa7837c8d6380ebdb in term 0.
I20260923 18:10:49.362356 29307 leader_election.cc:302] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba7a8e5d7ab48658931ef2c3f498575, f848ff76c9b6411fa7837c8d6380ebdb; no voters:
I20260923 18:10:49.362560 29596 raft_consensus.cc:2781] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:49.362630 29596 raft_consensus.cc:482] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:49.362674 29596 raft_consensus.cc:3037] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:49.363718 29596 raft_consensus.cc:504] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.364071 29596 leader_election.cc:288] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [CANDIDATE]: Term 1 election: Requested vote from peers 0ba7a8e5d7ab48658931ef2c3f498575 (127.27.75.193:38023), 5888b362069e4c3d83454faf5faf7f7e (127.27.75.195:46001)
I20260923 18:10:49.364450 29516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afda025f25f14ff597a9a55a4d8367b0" candidate_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5888b362069e4c3d83454faf5faf7f7e"
I20260923 18:10:49.364465 29235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afda025f25f14ff597a9a55a4d8367b0" candidate_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba7a8e5d7ab48658931ef2c3f498575"
I20260923 18:10:49.364562 29516 raft_consensus.cc:3037] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:49.364593 29235 raft_consensus.cc:3037] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:49.365576 29235 raft_consensus.cc:2454] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f848ff76c9b6411fa7837c8d6380ebdb in term 1.
I20260923 18:10:49.365600 29516 raft_consensus.cc:2454] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f848ff76c9b6411fa7837c8d6380ebdb in term 1.
I20260923 18:10:49.365883 29308 leader_election.cc:302] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5888b362069e4c3d83454faf5faf7f7e, f848ff76c9b6411fa7837c8d6380ebdb; no voters:
I20260923 18:10:49.366058 29596 raft_consensus.cc:2781] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:49.366139 29596 raft_consensus.cc:686] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [term 1 LEADER]: Becoming Leader. State: Replica: f848ff76c9b6411fa7837c8d6380ebdb, State: Running, Role: LEADER
I20260923 18:10:49.366276 29596 consensus_queue.cc:237] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } }
I20260923 18:10:49.366936 29592 tablet_replica.cc:439] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: TxnStatusTablet state changed. Reason: New leader f848ff76c9b6411fa7837c8d6380ebdb. Latest consensus state: current_term: 1 leader_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } } }
I20260923 18:10:49.367027 29592 tablet_replica.cc:442] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:10:49.367239 29627 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 18:10:49.367389 29627 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 18:10:49.368144 29076 catalog_manager.cc:5495] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb reported cstate change: term changed from 0 to 1, leader changed from <none> to f848ff76c9b6411fa7837c8d6380ebdb (127.27.75.194). New cstate: current_term: 1 leader_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } health_report { overall_health: UNKNOWN } } }
I20260923 18:10:49.391577 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.387586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:46071 (local address 127.0.0.1:47866)
0923 18:10:49.387843 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:49.387855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:49.387960 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:49.388360 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:49.388373 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:49.388382 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:10:49.388490 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.388495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.389588 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.389591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.390234 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.390245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.390842 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.390845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.390919 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.390973 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:10:49.391301 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:10:49.391310 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 18:10:49.391357 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 18:10:49.391443 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260923 18:10:49.391589 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.387696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.27.75.194:46071)
0923 18:10:49.387832 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:49.387837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:49.387910 (+ 73us) server_negotiation.cc:407] Connection header received
0923 18:10:49.388083 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:49.388086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:49.388147 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:49.388250 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:49.388603 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.389446 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.390371 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.390709 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.390810 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.391097 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:49.391197 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:49.391277 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:49.391385 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 18:10:49.391450 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:10:49.394379 29516 raft_consensus.cc:1261] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e [term 1 FOLLOWER]: Refusing update from remote peer f848ff76c9b6411fa7837c8d6380ebdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:49.394461 29235 raft_consensus.cc:1261] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 1 FOLLOWER]: Refusing update from remote peer f848ff76c9b6411fa7837c8d6380ebdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:49.394804 29596 consensus_queue.cc:1035] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:49.394968 29592 consensus_queue.cc:1035] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:49.396479 29593 tablet_replica.cc:439] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e: TxnStatusTablet state changed. Reason: New leader f848ff76c9b6411fa7837c8d6380ebdb. Latest consensus state: current_term: 1 leader_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } } }
I20260923 18:10:49.396605 29593 tablet_replica.cc:442] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:49.397732 29592 tablet_replica.cc:439] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } } }
I20260923 18:10:49.397847 29592 tablet_replica.cc:442] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:10:49.399297 29600 mvcc.cc:205] Tried to move back new op lower bound from 7332606154314936320 to 7332606154205704192. Current Snapshot: MvccSnapshot[applied={T|T < 7332606154314936320 or (T in {7332606154314936320})}]
I20260923 18:10:49.399456 29593 tablet_replica.cc:439] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } } }
I20260923 18:10:49.399573 29593 tablet_replica.cc:442] T afda025f25f14ff597a9a55a4d8367b0 P 5888b362069e4c3d83454faf5faf7f7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:49.402326 29594 tablet_replica.cc:439] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575: TxnStatusTablet state changed. Reason: New leader f848ff76c9b6411fa7837c8d6380ebdb. Latest consensus state: current_term: 1 leader_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } } }
I20260923 18:10:49.402455 29594 tablet_replica.cc:442] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:49.403152 29596 tablet_replica.cc:439] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } } }
I20260923 18:10:49.403275 29596 tablet_replica.cc:442] T afda025f25f14ff597a9a55a4d8367b0 P f848ff76c9b6411fa7837c8d6380ebdb: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:10:49.404314 29594 tablet_replica.cc:439] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 } } peers { permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 } } peers { permanent_uuid: "5888b362069e4c3d83454faf5faf7f7e" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46001 } } }
I20260923 18:10:49.404419 29594 tablet_replica.cc:442] T afda025f25f14ff597a9a55a4d8367b0 P 0ba7a8e5d7ab48658931ef2c3f498575: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:49.409327 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.405239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.27.75.195:46001)
0923 18:10:49.405355 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:49.405359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:49.405372 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:10:49.405446 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:49.405449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:49.405509 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:49.405602 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:49.405856 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.406453 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.407652 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.407935 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.408030 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.408357 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:49.408911 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:49.409019 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:49.409076 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 18:10:49.409159 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 18:10:49.420730 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.417002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:48355 (local address 127.27.75.194:46071)
0923 18:10:49.417154 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:49.417158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:49.417173 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:10:49.417276 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:49.417279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:49.417333 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:49.417401 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:49.417685 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.418508 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.419432 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.419774 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.419879 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.420205 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:49.420300 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:49.420368 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:49.420506 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 18:10:49.420568 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:10:49.420723 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.416869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:46071 (local address 127.27.75.195:48355)
0923 18:10:49.417027 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:49.417042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:49.417157 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:49.417445 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:49.417448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:49.417455 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:10:49.417583 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.417588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.418639 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.418642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.419318 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.419328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.419913 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.419917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.420019 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.420073 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:10:49.420409 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:10:49.420417 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 18:10:49.420486 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 18:10:49.420569 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 18:10:49.427536 29234 raft_consensus.cc:1261] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 1 FOLLOWER]: Refusing update from remote peer 5888b362069e4c3d83454faf5faf7f7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:49.427821 29375 raft_consensus.cc:1261] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb [term 1 FOLLOWER]: Refusing update from remote peer 5888b362069e4c3d83454faf5faf7f7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:49.428040 29615 consensus_queue.cc:1035] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba7a8e5d7ab48658931ef2c3f498575" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:49.428299 29593 consensus_queue.cc:1035] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f848ff76c9b6411fa7837c8d6380ebdb" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:49.435642 29601 mvcc.cc:205] Tried to move back new op lower bound from 7332606154450808832 to 7332606153666801664. Current Snapshot: MvccSnapshot[applied={T|T < 7332606154450808832}]
I20260923 18:10:49.450244 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.446687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:55609 (local address 127.27.75.195:46001)
0923 18:10:49.446808 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:49.446812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:49.447025 (+ 213us) server_negotiation.cc:407] Connection header received
0923 18:10:49.447154 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:49.447156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:49.447198 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:49.447259 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:49.447498 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.448054 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.449106 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.449389 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.449488 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.449815 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:49.449894 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:49.449953 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:49.450045 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 18:10:49.450102 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 18:10:49.450249 29639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.446546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:46001 (local address 127.27.75.194:55609)
0923 18:10:49.446935 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:49.446951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:49.447075 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:49.447284 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:49.447287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:49.447294 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:10:49.447415 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.447420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.448182 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.448185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.449008 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:49.449018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.449564 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.449566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:49.449631 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.449675 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:10:49.449969 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:10:49.449976 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:10:49.450034 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 18:10:49.450120 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260923 18:10:49.486399 29074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:49.486585 29074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:49.489084 29074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68c77781a41e4297a5d31563cecbaa72: Sending DeleteTablet for 3 replicas of tablet caf8414807904c3ea90eac648798caa7
I20260923 18:10:49.489784 29214 tablet_service.cc:1514] Processing DeleteTablet for tablet caf8414807904c3ea90eac648798caa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:49 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:40048
I20260923 18:10:49.489992 29610 tablet_replica.cc:330] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575: stopping tablet replica
I20260923 18:10:49.489988 29354 tablet_service.cc:1514] Processing DeleteTablet for tablet caf8414807904c3ea90eac648798caa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:49 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:47848
I20260923 18:10:49.490103 29610 raft_consensus.cc:2229] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:49.490239 29610 raft_consensus.cc:2258] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:49.490181 29609 tablet_replica.cc:330] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb: stopping tablet replica
I20260923 18:10:49.490397 29609 raft_consensus.cc:2229] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:49.490525 29609 raft_consensus.cc:2258] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:49.491020 29497 tablet_service.cc:1514] Processing DeleteTablet for tablet caf8414807904c3ea90eac648798caa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:49 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:44188
I20260923 18:10:49.491262 29608 tablet_replica.cc:330] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e: stopping tablet replica
I20260923 18:10:49.491377 29608 raft_consensus.cc:2229] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:10:49.491478 29610 ts_tablet_manager.cc:1905] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:49.491705 29608 raft_consensus.cc:2258] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:49.492602 29608 ts_tablet_manager.cc:1905] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:49.494580 29610 ts_tablet_manager.cc:1918] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:10:49.494655 29610 log.cc:1196] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-0/wal/wals/caf8414807904c3ea90eac648798caa7
I20260923 18:10:49.494886 29610 ts_tablet_manager.cc:1939] T caf8414807904c3ea90eac648798caa7 P 0ba7a8e5d7ab48658931ef2c3f498575: Deleting consensus metadata
I20260923 18:10:49.495667 29061 catalog_manager.cc:4841] TS 0ba7a8e5d7ab48658931ef2c3f498575 (127.27.75.193:38023): tablet caf8414807904c3ea90eac648798caa7 (table test-table [id=ccd36ba01fa54342b207f399bc21db5b]) successfully deleted
I20260923 18:10:49.496742 29609 ts_tablet_manager.cc:1905] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:49.497187 29608 ts_tablet_manager.cc:1918] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:10:49.497280 29608 log.cc:1196] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/caf8414807904c3ea90eac648798caa7
I20260923 18:10:49.497524 29608 ts_tablet_manager.cc:1939] T caf8414807904c3ea90eac648798caa7 P 5888b362069e4c3d83454faf5faf7f7e: Deleting consensus metadata
I20260923 18:10:49.498390 29062 catalog_manager.cc:4841] TS 5888b362069e4c3d83454faf5faf7f7e (127.27.75.195:46001): tablet caf8414807904c3ea90eac648798caa7 (table test-table [id=ccd36ba01fa54342b207f399bc21db5b]) successfully deleted
I20260923 18:10:49.499608 29609 ts_tablet_manager.cc:1918] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:10:49.499678 29609 log.cc:1196] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/caf8414807904c3ea90eac648798caa7
I20260923 18:10:49.499899 29609 ts_tablet_manager.cc:1939] T caf8414807904c3ea90eac648798caa7 P f848ff76c9b6411fa7837c8d6380ebdb: Deleting consensus metadata
I20260923 18:10:49.500687 29062 catalog_manager.cc:4841] TS f848ff76c9b6411fa7837c8d6380ebdb (127.27.75.194:46071): tablet caf8414807904c3ea90eac648798caa7 (table test-table [id=ccd36ba01fa54342b207f399bc21db5b]) successfully deleted
Sep 23 18:10:49 dist-test-slave-wsx5 krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:10:49.513705 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.501221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40056 (local address 127.27.75.193:38023)
0923 18:10:49.501358 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:49.501363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:49.501697 (+ 334us) server_negotiation.cc:407] Connection header received
0923 18:10:49.501821 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:49.501825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:49.501887 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:49.501982 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:49.502804 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.503609 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.504842 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.505222 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.505338 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.509307 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:49.509325 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:49.509345 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:49.509383 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:49.511619 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:49.512167 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:49.512174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:49.512176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:49.512290 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:49.512574 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:49.512577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:49.512579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:49.512915 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:49.513155 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:49.513373 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 18:10:49.513513 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260923 18:10:49.514180 29275 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40056
I20260923 18:10:49.525489 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:49.516144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.27.75.254:43703)
0923 18:10:49.516282 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:49.516286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:49.516521 (+ 235us) server_negotiation.cc:407] Connection header received
0923 18:10:49.516623 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:49.516626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:49.516686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:49.516771 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:49.517575 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.518400 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.519511 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:49.519844 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:49.519947 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:49.521719 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:49.521724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:49.521727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:49.521760 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:49.523640 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:49.524157 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:49.524161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:49.524162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:49.524255 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:49.524540 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:49.524543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:49.524545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:49.524774 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:49.524905 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:49.525176 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 18:10:49.525300 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 18:10:49.526206 29074 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53466
I20260923 18:10:49.527850 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 29149
I20260923 18:10:49.535559 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 29290
I20260923 18:10:49.542294 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 29431
I20260923 18:10:49.547610 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 29044
2026-09-23T18:10:49Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2647 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:49 dist-test-slave-wsx5 krb5kdc[29648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:49 dist-test-slave-wsx5 krb5kdc[29648](info): set up 2 sockets
Sep 23 18:10:49 dist-test-slave-wsx5 krb5kdc[29648](info): commencing operation
krb5kdc: starting...
W20260923 18:10:50.515043 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:10:50 dist-test-slave-wsx5 krb5kdc[29648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:10:50.636111 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39973
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:39973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:50.746861 29661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:50.747174 29661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:50.747221 29661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:50.751943 29661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:50.752028 29661 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 18:10:50.752049 29661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:50.752066 29661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:50.752081 29661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:50.752095 29661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:50.757893 29661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:39973
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:50.759382 29661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:50.760730 29661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:50.769980 29666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:50.770006 29667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:50.769970 29669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:50.770449 29661 server_base.cc:1034] running on GCE node
I20260923 18:10:50.771137 29661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:50.772452 29661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:50.773677 29661 hybrid_clock.cc:648] HybridClock initialized: now 1790187050773650 us; error 41 us; skew 500 ppm
Sep 23 18:10:50 dist-test-slave-wsx5 krb5kdc[29648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187050, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:50.776579 29661 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:10:50.777601 29661 webserver.cc:460] Webserver started at http://127.27.75.254:33301/ using document root <none> and password file <none>
I20260923 18:10:50.778151 29661 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:50.778218 29661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:50.778419 29661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:50.780476 29661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "2dbed242d40b4242bb41603fc98b454a"
format_stamp: "Formatted at 2026-09-23 18:10:50 on dist-test-slave-wsx5"
server_key: "19962d5aac7e7f64fac5c4190cd202fd"
server_key_iv: "5edea6f70b8e6939540b90f6dac9ce36"
server_key_version: "encryptionkey@0"
I20260923 18:10:50.781090 29661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "2dbed242d40b4242bb41603fc98b454a"
format_stamp: "Formatted at 2026-09-23 18:10:50 on dist-test-slave-wsx5"
server_key: "19962d5aac7e7f64fac5c4190cd202fd"
server_key_iv: "5edea6f70b8e6939540b90f6dac9ce36"
server_key_version: "encryptionkey@0"
I20260923 18:10:50.785012 29661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:10:50.787672 29676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:50.789016 29661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:10:50.789269 29661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "2dbed242d40b4242bb41603fc98b454a"
format_stamp: "Formatted at 2026-09-23 18:10:50 on dist-test-slave-wsx5"
server_key: "19962d5aac7e7f64fac5c4190cd202fd"
server_key_iv: "5edea6f70b8e6939540b90f6dac9ce36"
server_key_version: "encryptionkey@0"
I20260923 18:10:50.789383 29661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:50.806488 29661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:50.809743 29661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:50.809907 29661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:50.817796 29739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:39973 every 8 connection(s)
I20260923 18:10:50.817839 29661 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:39973
I20260923 18:10:50.818769 29661 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:50.821738 29740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:50.822577 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 29661
I20260923 18:10:50.822682 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:10:50.822943 27951 external_mini_cluster.cc:1428] Setting key 33bc07708654554ed0efee3326f828d7
I20260923 18:10:50.828109 29740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a: Bootstrap starting.
I20260923 18:10:50.830866 29740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:50.831758 29740 log.cc:824] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:10:50 dist-test-slave-wsx5 krb5kdc[29648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:50.834640 29740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a: No bootstrap required, opened a new log
I20260923 18:10:50.838680 29740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbed242d40b4242bb41603fc98b454a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39973 } }
I20260923 18:10:50.838992 29740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:50.839051 29740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dbed242d40b4242bb41603fc98b454a, State: Initialized, Role: FOLLOWER
I20260923 18:10:50.839443 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:50.824314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.27.75.254:39973)
0923 18:10:50.824715 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:50.824740 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:50.824787 (+ 47us) server_negotiation.cc:407] Connection header received
0923 18:10:50.825463 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:50.825484 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:50.825839 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:50.826132 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:50.827016 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:50.828015 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:50.829183 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:50.829671 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:50.829832 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:50.832815 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:50.832823 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:50.832842 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:50.832874 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:50.835668 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:50.836238 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:50.836245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:50.836251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:50.836412 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:50.836824 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:50.836828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:50.836830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:50.837445 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:50.837671 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:50.838022 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 18:10:50.838357 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":104,"threads_started":1}
I20260923 18:10:50.839545 29740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbed242d40b4242bb41603fc98b454a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39973 } }
I20260923 18:10:50.839696 29740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:50.839771 29740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:50.839884 29740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:50.841012 29740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbed242d40b4242bb41603fc98b454a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39973 } }
I20260923 18:10:50.841384 29740 leader_election.cc:302] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dbed242d40b4242bb41603fc98b454a; no voters:
I20260923 18:10:50.841722 29740 leader_election.cc:288] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:50.841943 29745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:50.842176 29745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [term 1 LEADER]: Becoming Leader. State: Replica: 2dbed242d40b4242bb41603fc98b454a, State: Running, Role: LEADER
I20260923 18:10:50.842501 29745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbed242d40b4242bb41603fc98b454a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39973 } }
I20260923 18:10:50.843021 29740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:50.844267 29746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dbed242d40b4242bb41603fc98b454a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbed242d40b4242bb41603fc98b454a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39973 } } }
I20260923 18:10:50.844297 29747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dbed242d40b4242bb41603fc98b454a. Latest consensus state: current_term: 1 leader_uuid: "2dbed242d40b4242bb41603fc98b454a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbed242d40b4242bb41603fc98b454a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39973 } } }
I20260923 18:10:50.844388 29746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:50.844394 29747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:50.844866 29754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:50.847999 29754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:50.854151 29754 catalog_manager.cc:1347] Generated new cluster ID: 289b82c26517439fa63daacc412a9b1a
I20260923 18:10:50.854246 29754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:50.881538 29754 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:50.882567 29754 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:50.891075 29754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a: Generated new TSK 0
I20260923 18:10:50.892035 29754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:10:50.929041 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39973
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:51.043262 29766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:51.043586 29766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:51.043646 29766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:51.048049 29766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:51.048123 29766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:51.048166 29766 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 18:10:51.048182 29766 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:10:51.048226 29766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:10:51.053699 29766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.75.254:39973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:51.054839 29766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:51.055887 29766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:51.065541 29772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:51.065584 29771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:51.065549 29774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:51.065938 29766 server_base.cc:1034] running on GCE node
I20260923 18:10:51.066323 29766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:51.066953 29766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:51.068128 29766 hybrid_clock.cc:648] HybridClock initialized: now 1790187051068093 us; error 48 us; skew 500 ppm
Sep 23 18:10:51 dist-test-slave-wsx5 krb5kdc[29648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187051, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:51.070911 29766 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:10:51.071780 29766 webserver.cc:460] Webserver started at http://127.27.75.193:34795/ using document root <none> and password file <none>
I20260923 18:10:51.072302 29766 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:51.072359 29766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:51.072522 29766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:51.074602 29766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "0711054c596345308af6abe05a50d169"
format_stamp: "Formatted at 2026-09-23 18:10:51 on dist-test-slave-wsx5"
server_key: "0f7ed70bb36ed78cb8eb8351a882d3c4"
server_key_iv: "98e922935809f15984b070f758cba234"
server_key_version: "encryptionkey@0"
I20260923 18:10:51.075155 29766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "0711054c596345308af6abe05a50d169"
format_stamp: "Formatted at 2026-09-23 18:10:51 on dist-test-slave-wsx5"
server_key: "0f7ed70bb36ed78cb8eb8351a882d3c4"
server_key_iv: "98e922935809f15984b070f758cba234"
server_key_version: "encryptionkey@0"
I20260923 18:10:51.078938 29766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:10:51.081372 29781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:51.082732 29766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:10:51.082881 29766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "0711054c596345308af6abe05a50d169"
format_stamp: "Formatted at 2026-09-23 18:10:51 on dist-test-slave-wsx5"
server_key: "0f7ed70bb36ed78cb8eb8351a882d3c4"
server_key_iv: "98e922935809f15984b070f758cba234"
server_key_version: "encryptionkey@0"
I20260923 18:10:51.082998 29766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:51.096095 29766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:51.099332 29766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:51.099511 29766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:51.101004 29766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:51.101056 29766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:51.101096 29766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:51.101111 29766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:10:51 dist-test-slave-wsx5 krb5kdc[29648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187051, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:51.111485 29766 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:41513
I20260923 18:10:51.111510 29897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:41513 every 8 connection(s)
I20260923 18:10:51.112550 29766 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:10:51.114982 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.101984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:54273 (local address 127.27.75.254:39973)
0923 18:10:51.102108 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.102112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.103227 (+ 1115us) server_negotiation.cc:407] Connection header received
0923 18:10:51.104333 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.104336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.104398 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.104476 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:51.106308 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.106917 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.108229 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.108490 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.108584 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.111271 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:51.111275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:51.111277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:51.111312 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:51.112821 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.113451 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.113454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.113456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.113538 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.113928 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.113930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.113932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.114121 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:51.114203 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.114711 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.114850 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 18:10:51.115388 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 29766
I20260923 18:10:51.115489 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:10:51.115747 27951 external_mini_cluster.cc:1428] Setting key 2554fd219944fda692c1a97b82a8f9ee
I20260923 18:10:51.115731 29790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.102519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39973 (local address 127.27.75.193:54273)
0923 18:10:51.103057 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.103102 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.104064 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.104735 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.104745 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.105318 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:51.106084 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.106109 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.107055 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.107058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.108098 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.108108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.108646 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.108650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.108829 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.109508 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:51.109530 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:51.111078 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:51.112943 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.112949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.112964 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.113330 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.113646 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.113649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.113652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.113820 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.114205 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:51.114210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:51.114444 (+ 234us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.114590 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.114851 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":155,"threads_started":1}
I20260923 18:10:51.117415 29898 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39973
I20260923 18:10:51.117570 29898 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:51.118144 29898 heartbeater.cc:511] Master 127.27.75.254:39973 requested a full tablet report, sending...
I20260923 18:10:51.119809 29693 ts_manager.cc:194] Registered new tserver with Master: 0711054c596345308af6abe05a50d169 (127.27.75.193:41513)
I20260923 18:10:51.121248 29693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:54273
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:10:51.126849 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.119764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:38531 (local address 127.27.75.254:39973)
0923 18:10:51.119875 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.119879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.119894 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:10:51.119993 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.119996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.120039 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.120088 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:51.120685 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.121531 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.122528 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.122764 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.122852 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.123905 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:51.123908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:51.123911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:51.123935 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:51.125335 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.125722 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.125725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.125727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.125805 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.126057 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.126060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.126062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.126240 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:51.126324 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.126570 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.126692 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
I20260923 18:10:51.126856 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.119508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39973 (local address 127.27.75.193:38531)
0923 18:10:51.119779 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.119793 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.119883 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.120232 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.120235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.120439 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:51.120558 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.120565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.121679 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.121682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.122416 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.122425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.122867 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.122871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.122958 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.123413 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:51.123425 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:51.123771 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:51.125434 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.125437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.125439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.125630 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.125893 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.125896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.125898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.125979 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.126328 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:51.126331 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:51.126422 (+ 91us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.126506 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.126658 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:10:51.142261 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39973
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:51.251974 29907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:51.252257 29907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:51.252310 29907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:51.256466 29907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:51.256542 29907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:51.256582 29907 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 18:10:51.256597 29907 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:10:51.256649 29907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:10:51.262283 29907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.75.254:39973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:51.263293 29907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:51.264261 29907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:51.273950 29912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:51.273953 29913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:51.273967 29907 server_base.cc:1034] running on GCE node
W20260923 18:10:51.273998 29915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:51.274653 29907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:51.275311 29907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:51.276494 29907 hybrid_clock.cc:648] HybridClock initialized: now 1790187051276473 us; error 37 us; skew 500 ppm
Sep 23 18:10:51 dist-test-slave-wsx5 krb5kdc[29648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187051, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:51.279198 29907 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:10:51.280066 29907 webserver.cc:460] Webserver started at http://127.27.75.194:36173/ using document root <none> and password file <none>
I20260923 18:10:51.280602 29907 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:51.280664 29907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:51.280895 29907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:51.282944 29907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "9c944ff15a46461dae9a892b5b9d6276"
format_stamp: "Formatted at 2026-09-23 18:10:51 on dist-test-slave-wsx5"
server_key: "b2dc88f8d03dba99cac50e1d801d2830"
server_key_iv: "accd596b90d054dc22adc3c5f82029d5"
server_key_version: "encryptionkey@0"
I20260923 18:10:51.283478 29907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "9c944ff15a46461dae9a892b5b9d6276"
format_stamp: "Formatted at 2026-09-23 18:10:51 on dist-test-slave-wsx5"
server_key: "b2dc88f8d03dba99cac50e1d801d2830"
server_key_iv: "accd596b90d054dc22adc3c5f82029d5"
server_key_version: "encryptionkey@0"
I20260923 18:10:51.287184 29907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 18:10:51.289600 29922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:51.290822 29907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:10:51.290925 29907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "9c944ff15a46461dae9a892b5b9d6276"
format_stamp: "Formatted at 2026-09-23 18:10:51 on dist-test-slave-wsx5"
server_key: "b2dc88f8d03dba99cac50e1d801d2830"
server_key_iv: "accd596b90d054dc22adc3c5f82029d5"
server_key_version: "encryptionkey@0"
I20260923 18:10:51.291033 29907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:51.318404 29907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:51.321617 29907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:51.321789 29907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:51.323254 29907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:51.323304 29907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:51.323359 29907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:51.323388 29907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:10:51 dist-test-slave-wsx5 krb5kdc[29648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187051, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:51.333220 29907 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:38129
I20260923 18:10:51.333318 30038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:38129 every 8 connection(s)
I20260923 18:10:51.334327 29907 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:10:51.336887 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.324249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:54595 (local address 127.27.75.254:39973)
0923 18:10:51.324382 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.324385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.325474 (+ 1089us) server_negotiation.cc:407] Connection header received
0923 18:10:51.326595 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.326599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.326665 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.326769 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:51.328290 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.329094 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.330384 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.330618 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.330719 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.333248 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:51.333252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:51.333255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:51.333289 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:51.334798 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.335430 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.335433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.335435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.335531 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.335949 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.335951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.335952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.336137 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:51.336221 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.336652 (+ 431us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.336766 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:10:51.337684 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.324753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39973 (local address 127.27.75.194:54595)
0923 18:10:51.325308 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.325355 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.326297 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.326964 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.326975 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.327416 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:51.328097 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.328119 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.329248 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.329252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.330255 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.330266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.330763 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.330767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.330953 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.331545 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:51.331565 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:51.333029 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:51.334937 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.334944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.334960 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.335335 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.335687 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.335690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.335692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.335860 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.336214 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:51.336218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:51.336443 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.336592 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.336854 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":141,"threads_started":1}
I20260923 18:10:51.338743 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 29907
I20260923 18:10:51.338837 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:10:51.339136 27951 external_mini_cluster.cc:1428] Setting key 98f6a2d2fa1790b3e0ef2437aa37021a
I20260923 18:10:51.341074 30039 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39973
I20260923 18:10:51.341238 30039 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:51.341888 30039 heartbeater.cc:511] Master 127.27.75.254:39973 requested a full tablet report, sending...
I20260923 18:10:51.342936 29692 ts_manager.cc:194] Registered new tserver with Master: 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129)
I20260923 18:10:51.343719 29692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:54595
I20260923 18:10:51.348984 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.341649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:57817 (local address 127.27.75.254:39973)
0923 18:10:51.341775 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.341779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.341789 (+ 10us) server_negotiation.cc:407] Connection header received
0923 18:10:51.341818 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.341820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.341853 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.341916 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:51.342517 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.343163 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.344120 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.344430 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.344501 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.346021 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:51.346024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:51.346026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:51.346047 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:51.347382 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.347850 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.347852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.347853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.347926 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.348182 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.348184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.348185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.348412 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:51.348484 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.348771 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.348865 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 18:10:51.349081 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.341249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39973 (local address 127.27.75.194:57817)
0923 18:10:51.341510 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.341526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.341620 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.342022 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.342024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.342241 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:51.342379 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.342387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.343303 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.343306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.343995 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.344007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.344647 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.344650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.344750 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.345412 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:51.345427 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:51.345883 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:51.347493 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.347496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.347499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.347761 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.348008 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.348011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.348013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.348107 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.348503 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:51.348506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:51.348613 (+ 107us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.348740 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.348894 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:10:51.369421 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39973
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:51.479326 30048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:51.479614 30048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:51.479662 30048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:51.483675 30048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:51.483749 30048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:51.483790 30048 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 18:10:51.483803 30048 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:10:51.483834 30048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:10:51.489220 30048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.75.254:39973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:51.490247 30048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:51.491200 30048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:51.500962 30056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:51.500972 30053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:51.500972 30054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:51.501051 30048 server_base.cc:1034] running on GCE node
I20260923 18:10:51.501701 30048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:51.502352 30048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:51.503552 30048 hybrid_clock.cc:648] HybridClock initialized: now 1790187051503529 us; error 37 us; skew 500 ppm
Sep 23 18:10:51 dist-test-slave-wsx5 krb5kdc[29648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187051, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:51.506441 30048 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:10:51.507447 30048 webserver.cc:460] Webserver started at http://127.27.75.195:43083/ using document root <none> and password file <none>
I20260923 18:10:51.508009 30048 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:51.508076 30048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:51.508286 30048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:51.510393 30048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "e492fe09301b4ae8962b2961838d2320"
format_stamp: "Formatted at 2026-09-23 18:10:51 on dist-test-slave-wsx5"
server_key: "03f8f949561c31bff4c34a0877030553"
server_key_iv: "78b4e21b78343b277557817033c855cd"
server_key_version: "encryptionkey@0"
I20260923 18:10:51.510947 30048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "e492fe09301b4ae8962b2961838d2320"
format_stamp: "Formatted at 2026-09-23 18:10:51 on dist-test-slave-wsx5"
server_key: "03f8f949561c31bff4c34a0877030553"
server_key_iv: "78b4e21b78343b277557817033c855cd"
server_key_version: "encryptionkey@0"
I20260923 18:10:51.514887 30048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:10:51.517917 30063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:51.519271 30048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:51.519387 30048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "e492fe09301b4ae8962b2961838d2320"
format_stamp: "Formatted at 2026-09-23 18:10:51 on dist-test-slave-wsx5"
server_key: "03f8f949561c31bff4c34a0877030553"
server_key_iv: "78b4e21b78343b277557817033c855cd"
server_key_version: "encryptionkey@0"
I20260923 18:10:51.519515 30048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:51.533963 30048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:51.537202 30048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:51.537385 30048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:51.538815 30048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:51.538862 30048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:51.538902 30048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:51.538929 30048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:10:51 dist-test-slave-wsx5 krb5kdc[29648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187051, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:51.548729 30048 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:35825
I20260923 18:10:51.548834 30179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:35825 every 8 connection(s)
I20260923 18:10:51.549790 30048 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:10:51.552493 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.539757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:59199 (local address 127.27.75.254:39973)
0923 18:10:51.539899 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.539904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.540802 (+ 898us) server_negotiation.cc:407] Connection header received
0923 18:10:51.542082 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.542085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.542149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.542254 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:51.543684 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.544558 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.545932 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.546257 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.546374 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.549041 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:51.549045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:51.549048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:51.549076 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:51.550671 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.551129 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.551132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.551134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.551211 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.551505 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.551508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.551510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.551685 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:51.551773 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.552223 (+ 450us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.552336 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:10:51.553288 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.540183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39973 (local address 127.27.75.195:59199)
0923 18:10:51.540633 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.540685 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.541874 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.542466 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.542473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.542932 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:51.543531 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.543548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.544695 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.544699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.545819 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.545830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.546430 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.546434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.546600 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.547275 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:51.547296 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:51.548825 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:51.550784 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.550790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.550804 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.551038 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.551292 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.551295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.551298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.551424 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.551784 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:51.551790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:51.552011 (+ 221us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.552168 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.552476 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260923 18:10:51.554484 30180 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39973
I20260923 18:10:51.554616 30180 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:51.555117 30180 heartbeater.cc:511] Master 127.27.75.254:39973 requested a full tablet report, sending...
I20260923 18:10:51.555850 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 30048
I20260923 18:10:51.555959 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:10:51.556159 29693 ts_manager.cc:194] Registered new tserver with Master: e492fe09301b4ae8962b2961838d2320 (127.27.75.195:35825)
I20260923 18:10:51.556200 27951 external_mini_cluster.cc:1428] Setting key 29d2d3637c361b95dee960225d292f79
I20260923 18:10:51.556819 29693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:59199
I20260923 18:10:51.558455 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:10:51.564724 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.556577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:53655 (local address 127.27.75.254:39973)
0923 18:10:51.556704 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.556708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.556931 (+ 223us) server_negotiation.cc:407] Connection header received
0923 18:10:51.557054 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.557056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.557101 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.557173 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:51.558204 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.558892 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.559677 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.559974 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.560073 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.561375 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:51.561378 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:51.561380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:51.561404 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:51.562956 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.563452 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.563457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.563459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.563556 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.563807 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.563810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.563811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.564048 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:51.564150 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.564492 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.564592 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:10:51.564822 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.556500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39973 (local address 127.27.75.195:53655)
0923 18:10:51.556849 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.556864 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.556954 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.557317 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.557319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.557530 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:51.557641 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.557648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.559012 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.559014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.559560 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.559567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.560119 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.560123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.560224 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.560760 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:51.560775 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:51.561227 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:51.563091 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.563094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.563097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.563323 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.563649 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.563651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.563653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.563728 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.564173 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:51.564177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:51.564286 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.564412 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.564583 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":85,"threads_started":1}
Sep 23 18:10:51 dist-test-slave-wsx5 krb5kdc[29648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 18:10:51 dist-test-slave-wsx5 krb5kdc[29648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:51.585918 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.577017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.27.75.254:39973)
0923 18:10:51.577180 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.577185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.577202 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:10:51.577245 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.577247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.577287 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.577363 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:51.578078 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.578622 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.579517 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.579803 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.579891 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.582472 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:51.582476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:51.582478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:51.582509 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:51.583838 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.584266 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.584269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.584270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.584364 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.584839 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.584842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.584843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.585050 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:51.585230 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.585615 (+ 385us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.585787 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:10:51.589250 29692 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:10:51.592271 29692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:51.607537 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.602410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.27.75.193:41513)
0923 18:10:51.602859 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.602875 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.602897 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:10:51.602980 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.602985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.603175 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.603341 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:51.603585 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.604379 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.606113 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.606709 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.606812 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.606992 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.607258 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.607365 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":92,"threads_started":1}
I20260923 18:10:51.607532 30199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.601486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:35825 (local address 127.0.0.1:44172)
0923 18:10:51.602011 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.602062 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.602197 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.603009 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.603015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.603038 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:51.603193 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.603200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.604800 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.604803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.606076 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.606085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.606845 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.606849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.607053 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.607102 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.607208 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.607368 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":115,"thread_start_us":79,"threads_started":1}
I20260923 18:10:51.607549 30197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.602222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:41513 (local address 127.0.0.1:36306)
0923 18:10:51.602722 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.602732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.602793 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.603328 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.603331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.603350 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:51.603462 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.603469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.604493 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.604497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.605960 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.605973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.606829 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.606832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.607059 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.607104 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.607211 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.607385 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":113,"thread_start_us":82,"threads_started":1}
I20260923 18:10:51.608160 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.601629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.27.75.195:35825)
0923 18:10:51.602488 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.602501 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.602525 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:10:51.602603 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.602609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.602842 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.603349 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:51.603494 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.604685 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.606191 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.606726 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.606827 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.606993 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.607901 (+ 908us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.608019 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":133,"threads_started":1}
I20260923 18:10:51.608572 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.602143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.27.75.194:38129)
0923 18:10:51.603044 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.603056 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.603077 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:10:51.603156 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.603161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.603356 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.603534 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:51.603752 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.604844 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.607065 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.607968 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.608074 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.608241 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.608351 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.608444 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":123,"threads_started":1}
I20260923 18:10:51.609561 30201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.601629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:38129 (local address 127.0.0.1:43324)
0923 18:10:51.602480 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.602493 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.602572 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.603513 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.603516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.603532 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:51.603634 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.603641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.604947 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.604951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.606070 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.606081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.608086 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.608090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.608204 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.608228 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.609346 (+ 1118us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.609426 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":41,"thread_start_us":140,"threads_started":1}
I20260923 18:10:51.610941 29831 tablet_service.cc:1467] Processing CreateTablet for tablet 7f33b054e19a40e7a46874cc3e8097e6 (DEFAULT_TABLE table=test-table [id=fada1c04a9b4432eb854db390a57a70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:51.610958 30112 tablet_service.cc:1467] Processing CreateTablet for tablet 7f33b054e19a40e7a46874cc3e8097e6 (DEFAULT_TABLE table=test-table [id=fada1c04a9b4432eb854db390a57a70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:51.611927 29831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f33b054e19a40e7a46874cc3e8097e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:51.611994 30112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f33b054e19a40e7a46874cc3e8097e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:51.612306 29972 tablet_service.cc:1467] Processing CreateTablet for tablet 7f33b054e19a40e7a46874cc3e8097e6 (DEFAULT_TABLE table=test-table [id=fada1c04a9b4432eb854db390a57a70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:51.613286 29972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f33b054e19a40e7a46874cc3e8097e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:51.619629 30203 tablet_bootstrap.cc:492] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: Bootstrap starting.
I20260923 18:10:51.620724 30205 tablet_bootstrap.cc:492] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: Bootstrap starting.
I20260923 18:10:51.621459 30204 tablet_bootstrap.cc:492] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: Bootstrap starting.
I20260923 18:10:51.621987 30203 tablet_bootstrap.cc:654] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:51.622910 30205 tablet_bootstrap.cc:654] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:51.622887 30203 log.cc:824] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:51.623219 30204 tablet_bootstrap.cc:654] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:51.623836 30205 log.cc:824] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:51.623973 30204 log.cc:824] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:51.625105 30203 tablet_bootstrap.cc:492] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: No bootstrap required, opened a new log
I20260923 18:10:51.625411 30203 ts_tablet_manager.cc:1397] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260923 18:10:51.625891 30205 tablet_bootstrap.cc:492] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: No bootstrap required, opened a new log
I20260923 18:10:51.626084 30205 ts_tablet_manager.cc:1397] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:10:51.626211 30204 tablet_bootstrap.cc:492] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: No bootstrap required, opened a new log
I20260923 18:10:51.626397 30204 ts_tablet_manager.cc:1397] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 18:10:51.629693 30204 raft_consensus.cc:348] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } }
I20260923 18:10:51.629709 30203 raft_consensus.cc:348] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } }
I20260923 18:10:51.630029 30203 raft_consensus.cc:374] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:51.630074 30204 raft_consensus.cc:374] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:51.630128 30203 raft_consensus.cc:729] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0711054c596345308af6abe05a50d169, State: Initialized, Role: FOLLOWER
I20260923 18:10:51.630144 30204 raft_consensus.cc:729] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e492fe09301b4ae8962b2961838d2320, State: Initialized, Role: FOLLOWER
I20260923 18:10:51.630137 30205 raft_consensus.cc:348] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } }
I20260923 18:10:51.630434 30205 raft_consensus.cc:374] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:51.630503 30205 raft_consensus.cc:729] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c944ff15a46461dae9a892b5b9d6276, State: Initialized, Role: FOLLOWER
I20260923 18:10:51.630592 30204 consensus_queue.cc:260] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } }
I20260923 18:10:51.630632 30203 consensus_queue.cc:260] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } }
I20260923 18:10:51.630913 30205 consensus_queue.cc:260] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } }
I20260923 18:10:51.631331 29898 heartbeater.cc:503] Master 127.27.75.254:39973 was elected leader, sending a full tablet report...
I20260923 18:10:51.631618 30039 heartbeater.cc:503] Master 127.27.75.254:39973 was elected leader, sending a full tablet report...
I20260923 18:10:51.631702 30204 ts_tablet_manager.cc:1428] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 18:10:51.632030 30180 heartbeater.cc:503] Master 127.27.75.254:39973 was elected leader, sending a full tablet report...
I20260923 18:10:51.633088 30203 ts_tablet_manager.cc:1428] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260923 18:10:51.634455 30205 ts_tablet_manager.cc:1428] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260923 18:10:51.689106 30209 raft_consensus.cc:482] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:51.689450 30209 raft_consensus.cc:504] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } }
I20260923 18:10:51.690691 30209 leader_election.cc:288] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e492fe09301b4ae8962b2961838d2320 (127.27.75.195:35825), 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129)
I20260923 18:10:51.694736 30212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.690950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:35825 (local address 127.27.75.193:34365)
0923 18:10:51.691240 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.691256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.691375 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.691633 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.691636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.691652 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:51.691809 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.691815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.692548 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.692552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.693646 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.693658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.694314 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.694318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.694408 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.694433 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.694498 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.694559 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":55,"spinlock_wait_cycles":15104,"thread_start_us":73,"threads_started":1}
I20260923 18:10:51.694747 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.691070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:34365 (local address 127.27.75.195:35825)
0923 18:10:51.691201 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.691206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.691337 (+ 131us) server_negotiation.cc:407] Connection header received
0923 18:10:51.691495 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.691498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.691550 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.691608 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:51.691908 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.692451 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.693786 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.694201 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.694310 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.694366 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.694529 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.694612 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:10:51.694834 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.691326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:49801 (local address 127.27.75.194:38129)
0923 18:10:51.691452 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.691456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.691470 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:10:51.691541 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.691543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.691578 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.691647 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:51.691952 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.692567 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.693774 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.694226 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.694338 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.694411 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.694619 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.694727 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 18:10:51.694784 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.690950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:38129 (local address 127.27.75.193:49801)
0923 18:10:51.691351 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.691364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.691462 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.691668 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.691671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.691712 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:51.691845 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.691850 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.692663 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.692666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.693665 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.693674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.694339 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.694342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.694496 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.694520 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.694591 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.694654 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":84,"spinlock_wait_cycles":3200,"thread_start_us":117,"threads_started":1}
I20260923 18:10:51.695830 30133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f33b054e19a40e7a46874cc3e8097e6" candidate_uuid: "0711054c596345308af6abe05a50d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e492fe09301b4ae8962b2961838d2320" is_pre_election: true
I20260923 18:10:51.695828 29993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f33b054e19a40e7a46874cc3e8097e6" candidate_uuid: "0711054c596345308af6abe05a50d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c944ff15a46461dae9a892b5b9d6276" is_pre_election: true
I20260923 18:10:51.696108 30133 raft_consensus.cc:2454] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0711054c596345308af6abe05a50d169 in term 0.
I20260923 18:10:51.696110 29993 raft_consensus.cc:2454] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0711054c596345308af6abe05a50d169 in term 0.
I20260923 18:10:51.696559 29785 leader_election.cc:302] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0711054c596345308af6abe05a50d169, e492fe09301b4ae8962b2961838d2320; no voters:
I20260923 18:10:51.696838 30209 raft_consensus.cc:2781] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:51.696929 30209 raft_consensus.cc:482] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:51.696975 30209 raft_consensus.cc:3037] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:51.698014 30209 raft_consensus.cc:504] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } }
I20260923 18:10:51.698335 30209 leader_election.cc:288] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [CANDIDATE]: Term 1 election: Requested vote from peers e492fe09301b4ae8962b2961838d2320 (127.27.75.195:35825), 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129)
I20260923 18:10:51.698701 30133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f33b054e19a40e7a46874cc3e8097e6" candidate_uuid: "0711054c596345308af6abe05a50d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e492fe09301b4ae8962b2961838d2320"
I20260923 18:10:51.698810 29993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f33b054e19a40e7a46874cc3e8097e6" candidate_uuid: "0711054c596345308af6abe05a50d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c944ff15a46461dae9a892b5b9d6276"
I20260923 18:10:51.698832 30133 raft_consensus.cc:3037] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:51.698935 29993 raft_consensus.cc:3037] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:51.699884 30133 raft_consensus.cc:2454] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0711054c596345308af6abe05a50d169 in term 1.
I20260923 18:10:51.699923 29993 raft_consensus.cc:2454] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0711054c596345308af6abe05a50d169 in term 1.
I20260923 18:10:51.700212 29785 leader_election.cc:302] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0711054c596345308af6abe05a50d169, e492fe09301b4ae8962b2961838d2320; no voters:
I20260923 18:10:51.700372 30209 raft_consensus.cc:2781] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:51.700567 30209 raft_consensus.cc:686] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 1 LEADER]: Becoming Leader. State: Replica: 0711054c596345308af6abe05a50d169, State: Running, Role: LEADER
I20260923 18:10:51.700934 30209 consensus_queue.cc:237] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } }
I20260923 18:10:51.704082 29692 catalog_manager.cc:5495] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0711054c596345308af6abe05a50d169 (127.27.75.193). New cstate: current_term: 1 leader_uuid: "0711054c596345308af6abe05a50d169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } health_report { overall_health: HEALTHY } } }
Sep 23 18:10:51 dist-test-slave-wsx5 krb5kdc[29648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187050, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:51.759159 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.750990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.27.75.254:39973)
0923 18:10:51.751114 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:51.751119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:51.751263 (+ 144us) server_negotiation.cc:407] Connection header received
0923 18:10:51.751391 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:51.751394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:51.751430 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:51.751492 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:51.752156 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.752680 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.753531 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.753745 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.753837 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.756239 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:51.756242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:51.756244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:51.756266 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:51.757569 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.757999 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.758002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.758004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.758073 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:51.758305 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:51.758308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:51.758310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:51.758487 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:51.758567 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:51.758858 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 18:10:51.759037 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":42}
I20260923 18:10:51.759161 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:51.750894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39973 (local address 127.0.0.1:47854)
0923 18:10:51.751177 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:51.751190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:51.751308 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:51.751523 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:51.751526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:51.751948 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:51.752045 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.752051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.752778 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.752781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.753423 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:51.753433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:51.753887 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:51.753891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:51.753971 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:51.754513 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:51.754525 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:51.756081 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:51.757667 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.757677 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.757684 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.757916 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.758148 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:51.758152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:51.758154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:51.758231 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:51.758580 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:51.758587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:51.758698 (+ 111us) client_negotiation.cc:769] Sending connection context
0923 18:10:51.758800 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 18:10:51.758967 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
I20260923 18:10:51.763387 29692 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:47854:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 18:10:51.764122 29692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:51.769026 29831 tablet_service.cc:1467] Processing CreateTablet for tablet c4366ae6532c47c29c59b2f58a7e24d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0fa58447f1e648919528c2939a443999]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:10:51.769496 29972 tablet_service.cc:1467] Processing CreateTablet for tablet c4366ae6532c47c29c59b2f58a7e24d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0fa58447f1e648919528c2939a443999]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:10:51.769894 29831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4366ae6532c47c29c59b2f58a7e24d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:51.769603 30112 tablet_service.cc:1467] Processing CreateTablet for tablet c4366ae6532c47c29c59b2f58a7e24d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0fa58447f1e648919528c2939a443999]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:10:51.770116 29972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4366ae6532c47c29c59b2f58a7e24d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:51.770200 30112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4366ae6532c47c29c59b2f58a7e24d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:51.772460 30205 tablet_bootstrap.cc:492] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276: Bootstrap starting.
I20260923 18:10:51.772819 30203 tablet_bootstrap.cc:492] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169: Bootstrap starting.
I20260923 18:10:51.773373 30204 tablet_bootstrap.cc:492] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: Bootstrap starting.
I20260923 18:10:51.773598 30205 tablet_bootstrap.cc:654] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:51.773994 30203 tablet_bootstrap.cc:654] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:51.774349 30204 tablet_bootstrap.cc:654] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:51.775130 30205 tablet_bootstrap.cc:492] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276: No bootstrap required, opened a new log
I20260923 18:10:51.775343 30205 ts_tablet_manager.cc:1397] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:10:51.775765 30203 tablet_bootstrap.cc:492] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169: No bootstrap required, opened a new log
I20260923 18:10:51.775853 30203 ts_tablet_manager.cc:1397] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260923 18:10:51.775866 30205 raft_consensus.cc:348] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:51.775986 30205 raft_consensus.cc:374] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:51.776024 30205 raft_consensus.cc:729] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c944ff15a46461dae9a892b5b9d6276, State: Initialized, Role: FOLLOWER
I20260923 18:10:51.776149 30205 consensus_queue.cc:260] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:51.776268 30203 raft_consensus.cc:348] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:51.776386 30203 raft_consensus.cc:374] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:51.776398 30205 ts_tablet_manager.cc:1428] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:10:51.776418 30203 raft_consensus.cc:729] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0711054c596345308af6abe05a50d169, State: Initialized, Role: FOLLOWER
I20260923 18:10:51.776453 30211 tablet_replica.cc:439] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } }
I20260923 18:10:51.776580 30211 tablet_replica.cc:442] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:51.776582 30203 consensus_queue.cc:260] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:51.776844 30203 ts_tablet_manager.cc:1428] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:10:51.776826 30209 tablet_replica.cc:439] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } }
I20260923 18:10:51.776944 30209 tablet_replica.cc:442] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:51.777617 30204 tablet_bootstrap.cc:492] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: No bootstrap required, opened a new log
I20260923 18:10:51.777710 30204 ts_tablet_manager.cc:1397] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 18:10:51.778096 30204 raft_consensus.cc:348] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:51.778194 30204 raft_consensus.cc:374] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:51.778214 30204 raft_consensus.cc:729] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e492fe09301b4ae8962b2961838d2320, State: Initialized, Role: FOLLOWER
I20260923 18:10:51.778317 30204 consensus_queue.cc:260] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:51.778561 30204 ts_tablet_manager.cc:1428] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:10:51.778569 30210 tablet_replica.cc:439] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } }
I20260923 18:10:51.778764 30210 tablet_replica.cc:442] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 18:10:51.800768 30181 tablet.cc:2369] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:10:51.835816 30040 tablet.cc:2369] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:10:51.864046 29899 tablet.cc:2369] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:10:52.000270 30210 raft_consensus.cc:482] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:52.000417 30210 raft_consensus.cc:504] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.001575 30210 leader_election.cc:288] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0711054c596345308af6abe05a50d169 (127.27.75.193:41513), 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129)
I20260923 18:10:52.005294 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.001825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:38129 (local address 127.27.75.195:41837)
0923 18:10:52.001940 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:52.001957 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:52.002052 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:52.002350 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:52.002353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:52.002374 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:52.002498 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.002504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.003290 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.003294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.004142 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.004152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.004834 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.004838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.004948 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.004973 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:10:52.005055 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 18:10:52.005166 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260923 18:10:52.005352 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.001919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:41837 (local address 127.27.75.194:38129)
0923 18:10:52.002035 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:52.002039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:52.002056 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:10:52.002176 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:52.002179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:52.002243 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:52.002333 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:52.002595 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.003177 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.004294 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.004709 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.004806 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.004848 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:52.005086 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 18:10:52.005182 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:10:52.005543 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.001826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:41513 (local address 127.27.75.195:49207)
0923 18:10:52.002258 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:52.002271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:52.002362 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:52.002577 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:52.002580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:52.002595 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:52.002709 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.002714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.003507 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.003510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.004296 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.004306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.005136 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.005141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.005231 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.005256 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:10:52.005337 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 18:10:52.005407 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":107,"thread_start_us":121,"threads_started":1}
I20260923 18:10:52.005561 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.001890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:49207 (local address 127.27.75.193:41513)
0923 18:10:52.002008 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:52.002012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:52.002355 (+ 343us) server_negotiation.cc:407] Connection header received
0923 18:10:52.002442 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:52.002445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:52.002491 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:52.002570 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:52.002851 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.003401 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.004466 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.004991 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.005106 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.005180 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:52.005368 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 18:10:52.005444 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 18:10:52.006055 29993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4366ae6532c47c29c59b2f58a7e24d7" candidate_uuid: "e492fe09301b4ae8962b2961838d2320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c944ff15a46461dae9a892b5b9d6276" is_pre_election: true
I20260923 18:10:52.006191 29993 raft_consensus.cc:2454] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e492fe09301b4ae8962b2961838d2320 in term 0.
I20260923 18:10:52.006199 29851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4366ae6532c47c29c59b2f58a7e24d7" candidate_uuid: "e492fe09301b4ae8962b2961838d2320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711054c596345308af6abe05a50d169" is_pre_election: true
I20260923 18:10:52.006444 29851 raft_consensus.cc:2454] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e492fe09301b4ae8962b2961838d2320 in term 0.
I20260923 18:10:52.006579 30064 leader_election.cc:302] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c944ff15a46461dae9a892b5b9d6276, e492fe09301b4ae8962b2961838d2320; no voters:
I20260923 18:10:52.006860 30210 raft_consensus.cc:2781] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:52.006915 30210 raft_consensus.cc:482] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:52.006942 30210 raft_consensus.cc:3037] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:52.007907 30210 raft_consensus.cc:504] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.008283 30210 leader_election.cc:288] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [CANDIDATE]: Term 1 election: Requested vote from peers 0711054c596345308af6abe05a50d169 (127.27.75.193:41513), 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129)
I20260923 18:10:52.008590 29851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4366ae6532c47c29c59b2f58a7e24d7" candidate_uuid: "e492fe09301b4ae8962b2961838d2320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711054c596345308af6abe05a50d169"
I20260923 18:10:52.008625 29993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4366ae6532c47c29c59b2f58a7e24d7" candidate_uuid: "e492fe09301b4ae8962b2961838d2320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c944ff15a46461dae9a892b5b9d6276"
I20260923 18:10:52.008718 29851 raft_consensus.cc:3037] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:52.008736 29993 raft_consensus.cc:3037] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:52.009851 29851 raft_consensus.cc:2454] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e492fe09301b4ae8962b2961838d2320 in term 1.
I20260923 18:10:52.009851 29993 raft_consensus.cc:2454] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e492fe09301b4ae8962b2961838d2320 in term 1.
I20260923 18:10:52.010126 30064 leader_election.cc:302] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c944ff15a46461dae9a892b5b9d6276, e492fe09301b4ae8962b2961838d2320; no voters:
I20260923 18:10:52.010344 30210 raft_consensus.cc:2781] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:52.010561 30210 raft_consensus.cc:686] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [term 1 LEADER]: Becoming Leader. State: Replica: e492fe09301b4ae8962b2961838d2320, State: Running, Role: LEADER
I20260923 18:10:52.010959 30210 consensus_queue.cc:237] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.013233 30226 tablet_replica.cc:439] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: TxnStatusTablet state changed. Reason: New leader e492fe09301b4ae8962b2961838d2320. Latest consensus state: current_term: 1 leader_uuid: "e492fe09301b4ae8962b2961838d2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } }
I20260923 18:10:52.013417 30226 tablet_replica.cc:442] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:10:52.013736 30228 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 18:10:52.013927 30228 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 18:10:52.014191 29693 catalog_manager.cc:5495] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 reported cstate change: term changed from 0 to 1, leader changed from <none> to e492fe09301b4ae8962b2961838d2320 (127.27.75.195). New cstate: current_term: 1 leader_uuid: "e492fe09301b4ae8962b2961838d2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } health_report { overall_health: HEALTHY } } }
I20260923 18:10:52.076907 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.072467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:35825 (local address 127.0.0.1:44178)
0923 18:10:52.072662 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:52.072678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:52.072796 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:52.073100 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:52.073102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:52.073111 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:10:52.073249 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.073255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.073998 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.074001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.074972 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.074983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.075526 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.075530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.075640 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.075715 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.076536 (+ 821us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.076557 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:10:52.076629 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:10:52.076743 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 18:10:52.076939 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.072609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.27.75.195:35825)
0923 18:10:52.072752 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:52.072755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:52.072769 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:10:52.072935 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:52.072937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:52.072997 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:52.073077 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:52.073345 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.073893 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.075148 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.075400 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.075507 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.075901 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.076437 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.076542 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:52.076751 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 18:10:52.076815 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 18:10:52.082322 29851 raft_consensus.cc:1261] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Refusing update from remote peer e492fe09301b4ae8962b2961838d2320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:52.082398 29993 raft_consensus.cc:1261] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Refusing update from remote peer e492fe09301b4ae8962b2961838d2320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:52.082959 30226 consensus_queue.cc:1035] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:52.083159 30210 consensus_queue.cc:1035] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:52.087028 30211 tablet_replica.cc:439] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276: TxnStatusTablet state changed. Reason: New leader e492fe09301b4ae8962b2961838d2320. Latest consensus state: current_term: 1 leader_uuid: "e492fe09301b4ae8962b2961838d2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } }
I20260923 18:10:52.087153 30211 tablet_replica.cc:442] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:52.087097 30209 tablet_replica.cc:439] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169: TxnStatusTablet state changed. Reason: New leader e492fe09301b4ae8962b2961838d2320. Latest consensus state: current_term: 1 leader_uuid: "e492fe09301b4ae8962b2961838d2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } }
I20260923 18:10:52.087195 30209 tablet_replica.cc:442] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:52.090924 30226 tablet_replica.cc:439] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e492fe09301b4ae8962b2961838d2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } }
I20260923 18:10:52.091048 30226 tablet_replica.cc:442] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:10:52.091488 30209 tablet_replica.cc:439] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e492fe09301b4ae8962b2961838d2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } }
I20260923 18:10:52.091595 30209 tablet_replica.cc:442] T c4366ae6532c47c29c59b2f58a7e24d7 P 0711054c596345308af6abe05a50d169: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:52.091998 30231 mvcc.cc:205] Tried to move back new op lower bound from 7332606165322063872 to 7332606165039534080. Current Snapshot: MvccSnapshot[applied={T|T < 7332606165322063872}]
I20260923 18:10:52.092129 30232 mvcc.cc:205] Tried to move back new op lower bound from 7332606165322063872 to 7332606165039534080. Current Snapshot: MvccSnapshot[applied={T|T < 7332606165322063872}]
I20260923 18:10:52.094153 30210 tablet_replica.cc:439] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e492fe09301b4ae8962b2961838d2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } }
I20260923 18:10:52.094267 30210 tablet_replica.cc:442] T c4366ae6532c47c29c59b2f58a7e24d7 P e492fe09301b4ae8962b2961838d2320: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:10:52.095125 30211 tablet_replica.cc:439] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e492fe09301b4ae8962b2961838d2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } } }
I20260923 18:10:52.095255 30211 tablet_replica.cc:442] T c4366ae6532c47c29c59b2f58a7e24d7 P 9c944ff15a46461dae9a892b5b9d6276: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:10:52.095888 30233 mvcc.cc:205] Tried to move back new op lower bound from 7332606165322063872 to 7332606165039534080. Current Snapshot: MvccSnapshot[applied={T|T < 7332606165322063872}]
I20260923 18:10:52.102646 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.098528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.27.75.193:41513)
0923 18:10:52.098637 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:52.098642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:52.098654 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:10:52.098895 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:52.098898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:52.098993 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:52.099093 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:52.099365 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.099935 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.101103 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.101385 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.101468 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.101846 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.102263 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.102354 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:52.102462 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 18:10:52.102514 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 18:10:52.114516 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.110414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:58747 (local address 127.27.75.195:35825)
0923 18:10:52.110601 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:52.110606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:52.110675 (+ 69us) server_negotiation.cc:407] Connection header received
0923 18:10:52.110827 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:52.110829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:52.110873 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:52.110933 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:52.111212 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.111963 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.113227 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.113553 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.113665 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.114060 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.114151 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.114225 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:52.114325 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 18:10:52.114382 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260923 18:10:52.114534 30245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.110280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:35825 (local address 127.27.75.193:58747)
0923 18:10:52.110592 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:52.110608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:52.110738 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:52.110965 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:52.110968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:52.110976 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:10:52.111126 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.111132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.112082 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.112085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.113094 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.113105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.113745 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.113748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.113853 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.113899 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.114236 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.114241 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 18:10:52.114306 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 18:10:52.114386 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
I20260923 18:10:52.122110 29993 raft_consensus.cc:1261] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Refusing update from remote peer 0711054c596345308af6abe05a50d169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:52.122480 30133 raft_consensus.cc:1261] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Refusing update from remote peer 0711054c596345308af6abe05a50d169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:52.122841 30209 consensus_queue.cc:1035] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:52.123448 30209 consensus_queue.cc:1035] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:52.149372 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.145534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:59617 (local address 127.27.75.193:41513)
0923 18:10:52.145647 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:52.145652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:52.145867 (+ 215us) server_negotiation.cc:407] Connection header received
0923 18:10:52.145984 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:52.145987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:52.146037 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:52.146096 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:52.146397 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.147158 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.148142 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.148480 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.148576 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.148858 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.148941 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.149002 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:52.149170 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 18:10:52.149228 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 18:10:52.149427 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.145454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:41513 (local address 127.27.75.195:59617)
0923 18:10:52.145786 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:52.145798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:52.145901 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:52.146112 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:52.146115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:52.146124 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:10:52.146254 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.146261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.147268 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.147272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.148030 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.148042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.148595 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.148598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.148664 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.148711 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.149051 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.149058 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:10:52.149159 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 18:10:52.149260 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":125,"threads_started":1}
I20260923 18:10:52.177922 29693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:52.178097 29693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:52.180581 29693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a: Sending DeleteTablet for 3 replicas of tablet 7f33b054e19a40e7a46874cc3e8097e6
I20260923 18:10:52.181407 29972 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f33b054e19a40e7a46874cc3e8097e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:52 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:43324
I20260923 18:10:52.181407 30112 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f33b054e19a40e7a46874cc3e8097e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:52 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:44172
I20260923 18:10:52.181948 30254 tablet_replica.cc:330] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: stopping tablet replica
I20260923 18:10:52.181912 29831 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f33b054e19a40e7a46874cc3e8097e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:52 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:36306
I20260923 18:10:52.182226 30253 tablet_replica.cc:330] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: stopping tablet replica
I20260923 18:10:52.182391 30255 tablet_replica.cc:330] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: stopping tablet replica
I20260923 18:10:52.182713 30255 raft_consensus.cc:2229] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:10:52.182755 29693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 18:10:52.183102 30255 raft_consensus.cc:2258] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 18:10:52.183193 29693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:52.183213 30253 raft_consensus.cc:2229] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:52.182719 30254 raft_consensus.cc:2229] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:52.183772 30254 raft_consensus.cc:2258] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:52.184310 30255 ts_tablet_manager.cc:1905] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:52.184602 30253 raft_consensus.cc:2258] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:52.186034 30253 ts_tablet_manager.cc:1905] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:52.187937 30254 ts_tablet_manager.cc:1905] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:52.188756 30112 tablet_service.cc:1467] Processing CreateTablet for tablet f80f21deaf3a442295e1a9c2de64f4a6 (DEFAULT_TABLE table=test-table [id=e13fce1b23544956b18707e20e64c243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:52.189015 30112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80f21deaf3a442295e1a9c2de64f4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:52.189571 30253 ts_tablet_manager.cc:1918] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:10:52.189756 30253 log.cc:1196] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/7f33b054e19a40e7a46874cc3e8097e6
I20260923 18:10:52.190120 30253 ts_tablet_manager.cc:1939] T 7f33b054e19a40e7a46874cc3e8097e6 P 9c944ff15a46461dae9a892b5b9d6276: Deleting consensus metadata
I20260923 18:10:52.190949 30254 ts_tablet_manager.cc:1918] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:10:52.191051 30254 log.cc:1196] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/7f33b054e19a40e7a46874cc3e8097e6
I20260923 18:10:52.191284 30204 tablet_bootstrap.cc:492] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320: Bootstrap starting.
I20260923 18:10:52.191368 30254 ts_tablet_manager.cc:1939] T 7f33b054e19a40e7a46874cc3e8097e6 P e492fe09301b4ae8962b2961838d2320: Deleting consensus metadata
I20260923 18:10:52.192163 30204 tablet_bootstrap.cc:654] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:52.192941 29677 catalog_manager.cc:4841] TS 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129): tablet 7f33b054e19a40e7a46874cc3e8097e6 (table test-table [id=fada1c04a9b4432eb854db390a57a70e]) successfully deleted
I20260923 18:10:52.193334 30204 tablet_bootstrap.cc:492] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320: No bootstrap required, opened a new log
I20260923 18:10:52.193411 30204 ts_tablet_manager.cc:1397] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:10:52.193405 30255 ts_tablet_manager.cc:1918] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:10:52.193516 30255 log.cc:1196] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/wal/wals/7f33b054e19a40e7a46874cc3e8097e6
I20260923 18:10:52.193729 29680 catalog_manager.cc:4841] TS e492fe09301b4ae8962b2961838d2320 (127.27.75.195:35825): tablet 7f33b054e19a40e7a46874cc3e8097e6 (table test-table [id=fada1c04a9b4432eb854db390a57a70e]) successfully deleted
I20260923 18:10:52.193914 30255 ts_tablet_manager.cc:1939] T 7f33b054e19a40e7a46874cc3e8097e6 P 0711054c596345308af6abe05a50d169: Deleting consensus metadata
I20260923 18:10:52.193878 30204 raft_consensus.cc:348] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.193988 30204 raft_consensus.cc:374] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:52.194017 30204 raft_consensus.cc:729] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e492fe09301b4ae8962b2961838d2320, State: Initialized, Role: FOLLOWER
I20260923 18:10:52.194133 30204 consensus_queue.cc:260] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.194382 30204 ts_tablet_manager.cc:1428] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:10:52.194491 29831 tablet_service.cc:1467] Processing CreateTablet for tablet f80f21deaf3a442295e1a9c2de64f4a6 (DEFAULT_TABLE table=test-table [id=e13fce1b23544956b18707e20e64c243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:52.194762 29831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80f21deaf3a442295e1a9c2de64f4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:52.195233 29679 catalog_manager.cc:4841] TS 0711054c596345308af6abe05a50d169 (127.27.75.193:41513): tablet 7f33b054e19a40e7a46874cc3e8097e6 (table test-table [id=fada1c04a9b4432eb854db390a57a70e]) successfully deleted
I20260923 18:10:52.196813 30203 tablet_bootstrap.cc:492] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169: Bootstrap starting.
I20260923 18:10:52.197042 29972 tablet_service.cc:1467] Processing CreateTablet for tablet f80f21deaf3a442295e1a9c2de64f4a6 (DEFAULT_TABLE table=test-table [id=e13fce1b23544956b18707e20e64c243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:52.197345 29972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80f21deaf3a442295e1a9c2de64f4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:52.197705 30203 tablet_bootstrap.cc:654] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:52.198925 30203 tablet_bootstrap.cc:492] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169: No bootstrap required, opened a new log
I20260923 18:10:52.199016 30203 ts_tablet_manager.cc:1397] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 18:10:52.199492 30203 raft_consensus.cc:348] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.199615 30203 raft_consensus.cc:374] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:52.199647 30203 raft_consensus.cc:729] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0711054c596345308af6abe05a50d169, State: Initialized, Role: FOLLOWER
I20260923 18:10:52.199654 30205 tablet_bootstrap.cc:492] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276: Bootstrap starting.
I20260923 18:10:52.199767 30203 consensus_queue.cc:260] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.200018 30203 ts_tablet_manager.cc:1428] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:10:52.200562 30205 tablet_bootstrap.cc:654] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:52.201987 30205 tablet_bootstrap.cc:492] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276: No bootstrap required, opened a new log
I20260923 18:10:52.202067 30205 ts_tablet_manager.cc:1397] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:10:52.202472 30205 raft_consensus.cc:348] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.202585 30205 raft_consensus.cc:374] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:52.202613 30205 raft_consensus.cc:729] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c944ff15a46461dae9a892b5b9d6276, State: Initialized, Role: FOLLOWER
I20260923 18:10:52.202762 30205 consensus_queue.cc:260] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.203003 30205 ts_tablet_manager.cc:1428] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:10:52.341794 30211 raft_consensus.cc:482] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:52.341931 30211 raft_consensus.cc:504] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.343108 30211 leader_election.cc:288] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0711054c596345308af6abe05a50d169 (127.27.75.193:41513), e492fe09301b4ae8962b2961838d2320 (127.27.75.195:35825)
I20260923 18:10:52.347828 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.343622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:58457 (local address 127.27.75.193:41513)
0923 18:10:52.343730 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:52.343734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:52.343785 (+ 51us) server_negotiation.cc:407] Connection header received
0923 18:10:52.343973 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:52.343977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:52.344044 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:52.344094 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:52.344541 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.345257 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.346601 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.347228 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.347351 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.347406 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:52.347603 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 18:10:52.347693 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 18:10:52.347931 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.343494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:43135 (local address 127.27.75.195:35825)
0923 18:10:52.343708 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:52.343713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:52.344449 (+ 736us) server_negotiation.cc:407] Connection header received
0923 18:10:52.344543 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:52.344546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:52.344600 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:52.344678 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:10:52.344891 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.345702 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.346995 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.347430 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.347547 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.347597 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:52.347759 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 18:10:52.347828 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260923 18:10:52.347954 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.343382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:35825 (local address 127.27.75.194:43135)
0923 18:10:52.344355 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:52.344368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:52.344458 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:52.344690 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:52.344693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:52.344710 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:52.344796 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.344802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.345811 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.345815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.346872 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.346880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.347545 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.347549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.347641 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.347662 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:10:52.347727 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 18:10:52.347791 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":375,"thread_start_us":81,"threads_started":1}
I20260923 18:10:52.348278 30259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.343378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:41513 (local address 127.27.75.194:58457)
0923 18:10:52.343686 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:52.343702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:52.343837 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:52.344241 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:52.344244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:52.344262 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:10:52.344402 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.344410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.345371 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.345373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.346472 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.346482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.347349 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.347353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.347451 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.347477 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:10:52.348038 (+ 561us) client_negotiation.cc:240] Negotiation successful
0923 18:10:52.348123 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260923 18:10:52.348712 29851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f21deaf3a442295e1a9c2de64f4a6" candidate_uuid: "9c944ff15a46461dae9a892b5b9d6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711054c596345308af6abe05a50d169" is_pre_election: true
I20260923 18:10:52.348785 30133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f21deaf3a442295e1a9c2de64f4a6" candidate_uuid: "9c944ff15a46461dae9a892b5b9d6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e492fe09301b4ae8962b2961838d2320" is_pre_election: true
I20260923 18:10:52.348879 29851 raft_consensus.cc:2454] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c944ff15a46461dae9a892b5b9d6276 in term 0.
I20260923 18:10:52.348891 30133 raft_consensus.cc:2454] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c944ff15a46461dae9a892b5b9d6276 in term 0.
I20260923 18:10:52.349287 29925 leader_election.cc:302] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0711054c596345308af6abe05a50d169, 9c944ff15a46461dae9a892b5b9d6276, e492fe09301b4ae8962b2961838d2320; no voters:
I20260923 18:10:52.349537 30211 raft_consensus.cc:2781] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:52.349597 30211 raft_consensus.cc:482] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:52.349637 30211 raft_consensus.cc:3037] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:52.350595 30211 raft_consensus.cc:504] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.350955 30211 leader_election.cc:288] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [CANDIDATE]: Term 1 election: Requested vote from peers 0711054c596345308af6abe05a50d169 (127.27.75.193:41513), e492fe09301b4ae8962b2961838d2320 (127.27.75.195:35825)
I20260923 18:10:52.351342 29851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f21deaf3a442295e1a9c2de64f4a6" candidate_uuid: "9c944ff15a46461dae9a892b5b9d6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711054c596345308af6abe05a50d169"
I20260923 18:10:52.351332 30133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f21deaf3a442295e1a9c2de64f4a6" candidate_uuid: "9c944ff15a46461dae9a892b5b9d6276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e492fe09301b4ae8962b2961838d2320"
I20260923 18:10:52.351476 29851 raft_consensus.cc:3037] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:52.351477 30133 raft_consensus.cc:3037] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:52.352631 30133 raft_consensus.cc:2454] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c944ff15a46461dae9a892b5b9d6276 in term 1.
I20260923 18:10:52.352632 29851 raft_consensus.cc:2454] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c944ff15a46461dae9a892b5b9d6276 in term 1.
I20260923 18:10:52.352952 29926 leader_election.cc:302] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c944ff15a46461dae9a892b5b9d6276, e492fe09301b4ae8962b2961838d2320; no voters:
I20260923 18:10:52.353171 30211 raft_consensus.cc:2781] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:52.353358 30211 raft_consensus.cc:686] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 LEADER]: Becoming Leader. State: Replica: 9c944ff15a46461dae9a892b5b9d6276, State: Running, Role: LEADER
I20260923 18:10:52.353588 30211 consensus_queue.cc:237] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:52.356406 29693 catalog_manager.cc:5495] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194). New cstate: current_term: 1 leader_uuid: "9c944ff15a46461dae9a892b5b9d6276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } health_report { overall_health: UNKNOWN } } }
I20260923 18:10:52.405855 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.402203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.27.75.194:38129)
0923 18:10:52.402321 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:52.402324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:52.402364 (+ 40us) server_negotiation.cc:407] Connection header received
0923 18:10:52.402550 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:52.402553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:52.402612 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:52.402706 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:52.402878 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.403388 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.404340 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.404557 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.404637 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.404907 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.405436 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.405522 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:52.405641 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 18:10:52.405693 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 18:10:52.416550 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.413006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:35825 (local address 127.27.75.194:58799)
0923 18:10:52.413325 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:52.413338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:52.413452 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:52.413773 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:52.413776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:52.413784 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:10:52.413903 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.413908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.414615 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.414619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.415253 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.415267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.415777 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.415780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.415858 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.415909 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.416266 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.416273 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:10:52.416336 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 18:10:52.416409 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":120,"threads_started":1}
I20260923 18:10:52.416560 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.413188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:58799 (local address 127.27.75.195:35825)
0923 18:10:52.413329 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:52.413333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:52.413397 (+ 64us) server_negotiation.cc:407] Connection header received
0923 18:10:52.413583 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:52.413586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:52.413651 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:52.413726 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:52.413998 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.414504 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.415382 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.415617 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.415739 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.416082 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.416162 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.416222 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:52.416389 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 18:10:52.416456 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:10:52.422665 30133 raft_consensus.cc:1261] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Refusing update from remote peer 9c944ff15a46461dae9a892b5b9d6276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:52.422648 29851 raft_consensus.cc:1261] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Refusing update from remote peer 9c944ff15a46461dae9a892b5b9d6276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:52.423166 30211 consensus_queue.cc:1035] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:52.423384 30261 consensus_queue.cc:1035] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:52.442488 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.439070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:38129 (local address 127.27.75.195:37809)
0923 18:10:52.439178 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:52.439191 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:52.439264 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:52.439599 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:52.439602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:52.439610 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:10:52.439712 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.439719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.440640 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.440642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.441296 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:52.441306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.441825 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.441829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:52.441924 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.441955 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.442223 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.442230 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:10:52.442286 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 18:10:52.442381 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 18:10:52.442479 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:52.439270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:37809 (local address 127.27.75.194:38129)
0923 18:10:52.439366 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:52.439371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:52.439400 (+ 29us) server_negotiation.cc:407] Connection header received
0923 18:10:52.439442 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:52.439445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:52.439496 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:52.439565 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:52.439826 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.440519 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.441378 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:52.441649 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:52.441809 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:52.442069 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:52.442144 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:52.442207 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:52.442297 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 18:10:52.442345 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260923 18:10:52.443351 30233 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 18:10:54.000088 29690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:54.000277 29690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:54.002933 29690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a: Sending DeleteTablet for 3 replicas of tablet f80f21deaf3a442295e1a9c2de64f4a6
I20260923 18:10:54.003546 29972 tablet_service.cc:1514] Processing DeleteTablet for tablet f80f21deaf3a442295e1a9c2de64f4a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:53 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:43324
I20260923 18:10:54.003551 29831 tablet_service.cc:1514] Processing DeleteTablet for tablet f80f21deaf3a442295e1a9c2de64f4a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:53 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:36306
I20260923 18:10:54.003597 30112 tablet_service.cc:1514] Processing DeleteTablet for tablet f80f21deaf3a442295e1a9c2de64f4a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:53 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:44172
I20260923 18:10:54.003976 30293 tablet_replica.cc:330] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276: stopping tablet replica
I20260923 18:10:54.004045 30295 tablet_replica.cc:330] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169: stopping tablet replica
I20260923 18:10:54.004117 30293 raft_consensus.cc:2229] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:10:54.004145 30295 raft_consensus.cc:2229] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:54.004316 30295 raft_consensus.cc:2258] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:54.004981 30295 ts_tablet_manager.cc:1905] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:54.005036 30293 raft_consensus.cc:2258] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:54.005612 30293 ts_tablet_manager.cc:1905] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:54.007761 30295 ts_tablet_manager.cc:1918] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 18:10:54.007846 30295 log.cc:1196] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/wal/wals/f80f21deaf3a442295e1a9c2de64f4a6
I20260923 18:10:54.008128 30295 ts_tablet_manager.cc:1939] T f80f21deaf3a442295e1a9c2de64f4a6 P 0711054c596345308af6abe05a50d169: Deleting consensus metadata
I20260923 18:10:54.008226 30293 ts_tablet_manager.cc:1918] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 18:10:54.008298 30293 log.cc:1196] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/f80f21deaf3a442295e1a9c2de64f4a6
I20260923 18:10:54.008540 30293 ts_tablet_manager.cc:1939] T f80f21deaf3a442295e1a9c2de64f4a6 P 9c944ff15a46461dae9a892b5b9d6276: Deleting consensus metadata
I20260923 18:10:54.008982 30294 tablet_replica.cc:330] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320: stopping tablet replica
I20260923 18:10:54.008975 29679 catalog_manager.cc:4841] TS 0711054c596345308af6abe05a50d169 (127.27.75.193:41513): tablet f80f21deaf3a442295e1a9c2de64f4a6 (table test-table [id=e13fce1b23544956b18707e20e64c243]) successfully deleted
I20260923 18:10:54.009155 30294 raft_consensus.cc:2229] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:54.009219 29677 catalog_manager.cc:4841] TS 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129): tablet f80f21deaf3a442295e1a9c2de64f4a6 (table test-table [id=e13fce1b23544956b18707e20e64c243]) successfully deleted
I20260923 18:10:54.009313 30294 raft_consensus.cc:2258] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:54.009867 30294 ts_tablet_manager.cc:1905] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:54.011965 30294 ts_tablet_manager.cc:1918] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 18:10:54.012036 30294 log.cc:1196] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/f80f21deaf3a442295e1a9c2de64f4a6
I20260923 18:10:54.012245 30294 ts_tablet_manager.cc:1939] T f80f21deaf3a442295e1a9c2de64f4a6 P e492fe09301b4ae8962b2961838d2320: Deleting consensus metadata
I20260923 18:10:54.012948 29680 catalog_manager.cc:4841] TS e492fe09301b4ae8962b2961838d2320 (127.27.75.195:35825): tablet f80f21deaf3a442295e1a9c2de64f4a6 (table test-table [id=e13fce1b23544956b18707e20e64c243]) successfully deleted
Sep 23 18:10:54 dist-test-slave-wsx5 krb5kdc[29648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 18:10:54 dist-test-slave-wsx5 krb5kdc[29648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:54.032644 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:54.023738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.27.75.254:39973)
0923 18:10:54.024041 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:54.024049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:54.024174 (+ 125us) server_negotiation.cc:407] Connection header received
0923 18:10:54.024313 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:54.024316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:54.024366 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:54.024441 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:54.025181 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:54.025749 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:54.026653 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:54.026887 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:54.026970 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:54.029387 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:54.029391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:54.029392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:54.029434 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:54.030898 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:54.031337 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:54.031340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:54.031341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:54.031433 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:54.031708 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:54.031711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:54.031712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:54.031920 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:54.032026 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:54.032312 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 18:10:54.032457 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260923 18:10:54.035322 29690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:10:54.035732 29690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:54.040815 29831 tablet_service.cc:1467] Processing CreateTablet for tablet 6aae028fa60648c886ef8aaa9700bed2 (DEFAULT_TABLE table=test-table [id=d841dac47a5e4e0d84fc8e1e62172174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:54.040901 30112 tablet_service.cc:1467] Processing CreateTablet for tablet 6aae028fa60648c886ef8aaa9700bed2 (DEFAULT_TABLE table=test-table [id=d841dac47a5e4e0d84fc8e1e62172174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:54.040971 29972 tablet_service.cc:1467] Processing CreateTablet for tablet 6aae028fa60648c886ef8aaa9700bed2 (DEFAULT_TABLE table=test-table [id=d841dac47a5e4e0d84fc8e1e62172174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:54.041172 29831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aae028fa60648c886ef8aaa9700bed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:54.041177 30112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aae028fa60648c886ef8aaa9700bed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:54.041311 29972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aae028fa60648c886ef8aaa9700bed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:54.043424 30304 tablet_bootstrap.cc:492] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276: Bootstrap starting.
I20260923 18:10:54.044111 30306 tablet_bootstrap.cc:492] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320: Bootstrap starting.
I20260923 18:10:54.044510 30304 tablet_bootstrap.cc:654] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:54.044924 30306 tablet_bootstrap.cc:654] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:54.045665 30304 tablet_bootstrap.cc:492] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276: No bootstrap required, opened a new log
I20260923 18:10:54.045742 30304 ts_tablet_manager.cc:1397] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:10:54.045981 30306 tablet_bootstrap.cc:492] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320: No bootstrap required, opened a new log
I20260923 18:10:54.046060 30306 ts_tablet_manager.cc:1397] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 18:10:54.046103 30304 raft_consensus.cc:348] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.046228 30304 raft_consensus.cc:374] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:54.046259 30304 raft_consensus.cc:729] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c944ff15a46461dae9a892b5b9d6276, State: Initialized, Role: FOLLOWER
I20260923 18:10:54.046409 30304 consensus_queue.cc:260] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.046416 30306 raft_consensus.cc:348] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.046518 30306 raft_consensus.cc:374] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:54.046558 30306 raft_consensus.cc:729] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e492fe09301b4ae8962b2961838d2320, State: Initialized, Role: FOLLOWER
I20260923 18:10:54.046679 30304 ts_tablet_manager.cc:1428] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:10:54.046696 30306 consensus_queue.cc:260] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.046828 30305 tablet_bootstrap.cc:492] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169: Bootstrap starting.
I20260923 18:10:54.047040 30306 ts_tablet_manager.cc:1428] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 18:10:54.047674 30305 tablet_bootstrap.cc:654] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:54.049019 30305 tablet_bootstrap.cc:492] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169: No bootstrap required, opened a new log
I20260923 18:10:54.049185 30305 ts_tablet_manager.cc:1397] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 18:10:54.049659 30305 raft_consensus.cc:348] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.049780 30305 raft_consensus.cc:374] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:54.049823 30305 raft_consensus.cc:729] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0711054c596345308af6abe05a50d169, State: Initialized, Role: FOLLOWER
I20260923 18:10:54.049945 30305 consensus_queue.cc:260] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.050251 30305 ts_tablet_manager.cc:1428] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:10:54.137216 30274 raft_consensus.cc:482] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:54.137369 30274 raft_consensus.cc:504] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.137874 30274 leader_election.cc:288] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0711054c596345308af6abe05a50d169 (127.27.75.193:41513), 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129)
I20260923 18:10:54.138478 29850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aae028fa60648c886ef8aaa9700bed2" candidate_uuid: "e492fe09301b4ae8962b2961838d2320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711054c596345308af6abe05a50d169" is_pre_election: true
I20260923 18:10:54.138477 29993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aae028fa60648c886ef8aaa9700bed2" candidate_uuid: "e492fe09301b4ae8962b2961838d2320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c944ff15a46461dae9a892b5b9d6276" is_pre_election: true
I20260923 18:10:54.138641 29993 raft_consensus.cc:2454] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e492fe09301b4ae8962b2961838d2320 in term 0.
I20260923 18:10:54.138641 29850 raft_consensus.cc:2454] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e492fe09301b4ae8962b2961838d2320 in term 0.
I20260923 18:10:54.139046 30066 leader_election.cc:302] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0711054c596345308af6abe05a50d169, e492fe09301b4ae8962b2961838d2320; no voters:
I20260923 18:10:54.139232 30274 raft_consensus.cc:2781] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:54.139309 30274 raft_consensus.cc:482] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:54.139351 30274 raft_consensus.cc:3037] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:54.140527 30274 raft_consensus.cc:504] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.140911 30274 leader_election.cc:288] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [CANDIDATE]: Term 1 election: Requested vote from peers 0711054c596345308af6abe05a50d169 (127.27.75.193:41513), 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129)
I20260923 18:10:54.141348 29993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aae028fa60648c886ef8aaa9700bed2" candidate_uuid: "e492fe09301b4ae8962b2961838d2320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c944ff15a46461dae9a892b5b9d6276"
I20260923 18:10:54.141351 29850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aae028fa60648c886ef8aaa9700bed2" candidate_uuid: "e492fe09301b4ae8962b2961838d2320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711054c596345308af6abe05a50d169"
I20260923 18:10:54.141479 29850 raft_consensus.cc:3037] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:54.141487 29993 raft_consensus.cc:3037] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:54.142519 29850 raft_consensus.cc:2454] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e492fe09301b4ae8962b2961838d2320 in term 1.
I20260923 18:10:54.142537 29993 raft_consensus.cc:2454] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e492fe09301b4ae8962b2961838d2320 in term 1.
I20260923 18:10:54.142848 30066 leader_election.cc:302] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0711054c596345308af6abe05a50d169, e492fe09301b4ae8962b2961838d2320; no voters:
I20260923 18:10:54.143026 30274 raft_consensus.cc:2781] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:54.143123 30274 raft_consensus.cc:686] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 1 LEADER]: Becoming Leader. State: Replica: e492fe09301b4ae8962b2961838d2320, State: Running, Role: LEADER
I20260923 18:10:54.143260 30274 consensus_queue.cc:237] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.145032 29690 catalog_manager.cc:5495] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 reported cstate change: term changed from 0 to 1, leader changed from <none> to e492fe09301b4ae8962b2961838d2320 (127.27.75.195). New cstate: current_term: 1 leader_uuid: "e492fe09301b4ae8962b2961838d2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } health_report { overall_health: HEALTHY } } }
I20260923 18:10:54.180384 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:54.177283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.27.75.195:35825)
0923 18:10:54.177538 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:54.177544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:54.177555 (+ 11us) server_negotiation.cc:407] Connection header received
0923 18:10:54.177593 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:54.177595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:54.177636 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:54.177756 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:54.178048 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:54.178579 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:54.179446 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:54.179717 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:54.179798 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:54.179991 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:54.180059 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:54.180106 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:54.180175 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 18:10:54.180228 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":109,"threads_started":1}
I20260923 18:10:54.189529 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:54.186493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:39265 (local address 127.27.75.195:35825)
0923 18:10:54.186599 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:54.186602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:54.186663 (+ 61us) server_negotiation.cc:407] Connection header received
0923 18:10:54.186803 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:54.186805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:54.186847 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:54.186911 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:54.187109 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:54.187661 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:54.188464 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:54.188709 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:54.188801 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:54.189046 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:54.189162 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:54.189236 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:54.189323 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 18:10:54.189371 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":44}
I20260923 18:10:54.189538 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:54.186321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:35825 (local address 127.27.75.195:39265)
0923 18:10:54.186588 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:54.186601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:54.186688 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:54.186928 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:54.186930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:54.186936 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:10:54.187035 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:54.187040 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:54.187749 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:54.187752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:54.188352 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:54.188360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:54.188826 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:54.188829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:54.188897 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:54.188923 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:10:54.189247 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:10:54.189254 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:10:54.189308 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 18:10:54.189378 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260923 18:10:54.194888 29993 raft_consensus.cc:1261] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Refusing update from remote peer e492fe09301b4ae8962b2961838d2320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:54.194888 29850 raft_consensus.cc:1261] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Refusing update from remote peer e492fe09301b4ae8962b2961838d2320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:54.195241 30274 consensus_queue.cc:1035] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:54.195374 30288 consensus_queue.cc:1035] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:54.227140 29690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:54.227294 29690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:54.229519 29690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a: Sending DeleteTablet for 3 replicas of tablet 6aae028fa60648c886ef8aaa9700bed2
I20260923 18:10:54.230075 30112 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aae028fa60648c886ef8aaa9700bed2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:54 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:44172
I20260923 18:10:54.230098 29831 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aae028fa60648c886ef8aaa9700bed2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:54 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:36306
I20260923 18:10:54.230158 29972 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aae028fa60648c886ef8aaa9700bed2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:54 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:43324
I20260923 18:10:54.230379 30294 tablet_replica.cc:330] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320: stopping tablet replica
I20260923 18:10:54.230379 30293 tablet_replica.cc:330] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276: stopping tablet replica
I20260923 18:10:54.230429 30295 tablet_replica.cc:330] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169: stopping tablet replica
I20260923 18:10:54.230500 30293 raft_consensus.cc:2229] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:54.230500 30294 raft_consensus.cc:2229] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:10:54.230530 30295 raft_consensus.cc:2229] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:54.230615 30293 raft_consensus.cc:2258] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:54.230628 30295 raft_consensus.cc:2258] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:54.230814 30294 raft_consensus.cc:2258] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:54.231314 30295 ts_tablet_manager.cc:1905] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:54.231482 30293 ts_tablet_manager.cc:1905] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:54.231549 30294 ts_tablet_manager.cc:1905] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:54.233994 30293 ts_tablet_manager.cc:1918] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:10:54.233994 30295 ts_tablet_manager.cc:1918] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:10:54.234064 30293 log.cc:1196] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/6aae028fa60648c886ef8aaa9700bed2
I20260923 18:10:54.234064 30295 log.cc:1196] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-0/wal/wals/6aae028fa60648c886ef8aaa9700bed2
I20260923 18:10:54.234303 30295 ts_tablet_manager.cc:1939] T 6aae028fa60648c886ef8aaa9700bed2 P 0711054c596345308af6abe05a50d169: Deleting consensus metadata
I20260923 18:10:54.234321 30293 ts_tablet_manager.cc:1939] T 6aae028fa60648c886ef8aaa9700bed2 P 9c944ff15a46461dae9a892b5b9d6276: Deleting consensus metadata
I20260923 18:10:54.235113 29679 catalog_manager.cc:4841] TS 0711054c596345308af6abe05a50d169 (127.27.75.193:41513): tablet 6aae028fa60648c886ef8aaa9700bed2 (table test-table [id=d841dac47a5e4e0d84fc8e1e62172174]) successfully deleted
I20260923 18:10:54.235186 30294 ts_tablet_manager.cc:1918] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:10:54.235265 30294 log.cc:1196] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/6aae028fa60648c886ef8aaa9700bed2
I20260923 18:10:54.235494 30294 ts_tablet_manager.cc:1939] T 6aae028fa60648c886ef8aaa9700bed2 P e492fe09301b4ae8962b2961838d2320: Deleting consensus metadata
I20260923 18:10:54.235471 29690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:10:54.235931 29690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:10:54.236089 29677 catalog_manager.cc:4841] TS 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129): tablet 6aae028fa60648c886ef8aaa9700bed2 (table test-table [id=d841dac47a5e4e0d84fc8e1e62172174]) successfully deleted
I20260923 18:10:54.236125 29680 catalog_manager.cc:4841] TS e492fe09301b4ae8962b2961838d2320 (127.27.75.195:35825): tablet 6aae028fa60648c886ef8aaa9700bed2 (table test-table [id=d841dac47a5e4e0d84fc8e1e62172174]) successfully deleted
I20260923 18:10:54.241101 29972 tablet_service.cc:1467] Processing CreateTablet for tablet a612657fc6744bd885a35f1f52b6f4e2 (DEFAULT_TABLE table=test-table [id=6a468e9a9a3b4da2ad6112add1c6da28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:54.241281 30112 tablet_service.cc:1467] Processing CreateTablet for tablet a612657fc6744bd885a35f1f52b6f4e2 (DEFAULT_TABLE table=test-table [id=6a468e9a9a3b4da2ad6112add1c6da28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:54.241426 29972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a612657fc6744bd885a35f1f52b6f4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:54.241542 30112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a612657fc6744bd885a35f1f52b6f4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:54.243377 30304 tablet_bootstrap.cc:492] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276: Bootstrap starting.
I20260923 18:10:54.243577 29831 tablet_service.cc:1467] Processing CreateTablet for tablet a612657fc6744bd885a35f1f52b6f4e2 (DEFAULT_TABLE table=test-table [id=6a468e9a9a3b4da2ad6112add1c6da28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:10:54.243825 29831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a612657fc6744bd885a35f1f52b6f4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:54.244233 30304 tablet_bootstrap.cc:654] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:54.244771 30306 tablet_bootstrap.cc:492] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320: Bootstrap starting.
I20260923 18:10:54.245603 30306 tablet_bootstrap.cc:654] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:54.246186 30305 tablet_bootstrap.cc:492] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169: Bootstrap starting.
I20260923 18:10:54.246575 30304 tablet_bootstrap.cc:492] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276: No bootstrap required, opened a new log
I20260923 18:10:54.246651 30304 ts_tablet_manager.cc:1397] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 18:10:54.246949 30305 tablet_bootstrap.cc:654] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:54.247044 30304 raft_consensus.cc:348] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.247153 30304 raft_consensus.cc:374] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:54.247184 30304 raft_consensus.cc:729] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c944ff15a46461dae9a892b5b9d6276, State: Initialized, Role: FOLLOWER
I20260923 18:10:54.247324 30304 consensus_queue.cc:260] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.247586 30304 ts_tablet_manager.cc:1428] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:10:54.248113 30305 tablet_bootstrap.cc:492] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169: No bootstrap required, opened a new log
I20260923 18:10:54.248189 30305 ts_tablet_manager.cc:1397] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 18:10:54.248588 30305 raft_consensus.cc:348] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.248675 30306 tablet_bootstrap.cc:492] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320: No bootstrap required, opened a new log
I20260923 18:10:54.248692 30305 raft_consensus.cc:374] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:54.248720 30305 raft_consensus.cc:729] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0711054c596345308af6abe05a50d169, State: Initialized, Role: FOLLOWER
I20260923 18:10:54.248749 30306 ts_tablet_manager.cc:1397] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 18:10:54.248843 30305 consensus_queue.cc:260] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.249081 30305 ts_tablet_manager.cc:1428] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:10:54.249495 30306 raft_consensus.cc:348] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.249622 30306 raft_consensus.cc:374] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:54.249665 30306 raft_consensus.cc:729] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e492fe09301b4ae8962b2961838d2320, State: Initialized, Role: FOLLOWER
I20260923 18:10:54.249787 30306 consensus_queue.cc:260] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.250028 30306 ts_tablet_manager.cc:1428] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:10:54.422919 30310 raft_consensus.cc:482] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:10:54.423080 30310 raft_consensus.cc:504] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.423527 30310 leader_election.cc:288] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129), e492fe09301b4ae8962b2961838d2320 (127.27.75.195:35825)
I20260923 18:10:54.424086 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a612657fc6744bd885a35f1f52b6f4e2" candidate_uuid: "0711054c596345308af6abe05a50d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e492fe09301b4ae8962b2961838d2320" is_pre_election: true
I20260923 18:10:54.424136 29993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a612657fc6744bd885a35f1f52b6f4e2" candidate_uuid: "0711054c596345308af6abe05a50d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c944ff15a46461dae9a892b5b9d6276" is_pre_election: true
I20260923 18:10:54.424259 30132 raft_consensus.cc:2454] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0711054c596345308af6abe05a50d169 in term 0.
I20260923 18:10:54.424307 29993 raft_consensus.cc:2454] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0711054c596345308af6abe05a50d169 in term 0.
I20260923 18:10:54.424585 29785 leader_election.cc:302] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0711054c596345308af6abe05a50d169, e492fe09301b4ae8962b2961838d2320; no voters:
I20260923 18:10:54.424789 30310 raft_consensus.cc:2781] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:10:54.424847 30310 raft_consensus.cc:482] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:10:54.424872 30310 raft_consensus.cc:3037] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:54.425962 30310 raft_consensus.cc:504] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.426285 30310 leader_election.cc:288] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [CANDIDATE]: Term 1 election: Requested vote from peers 9c944ff15a46461dae9a892b5b9d6276 (127.27.75.194:38129), e492fe09301b4ae8962b2961838d2320 (127.27.75.195:35825)
I20260923 18:10:54.426703 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a612657fc6744bd885a35f1f52b6f4e2" candidate_uuid: "0711054c596345308af6abe05a50d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e492fe09301b4ae8962b2961838d2320"
I20260923 18:10:54.426734 29993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a612657fc6744bd885a35f1f52b6f4e2" candidate_uuid: "0711054c596345308af6abe05a50d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c944ff15a46461dae9a892b5b9d6276"
I20260923 18:10:54.426843 29993 raft_consensus.cc:3037] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:54.426843 30132 raft_consensus.cc:3037] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:54.427990 30132 raft_consensus.cc:2454] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0711054c596345308af6abe05a50d169 in term 1.
I20260923 18:10:54.427990 29993 raft_consensus.cc:2454] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0711054c596345308af6abe05a50d169 in term 1.
I20260923 18:10:54.428301 29785 leader_election.cc:302] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0711054c596345308af6abe05a50d169, e492fe09301b4ae8962b2961838d2320; no voters:
I20260923 18:10:54.428503 30310 raft_consensus.cc:2781] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:54.428679 30310 raft_consensus.cc:686] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [term 1 LEADER]: Becoming Leader. State: Replica: 0711054c596345308af6abe05a50d169, State: Running, Role: LEADER
I20260923 18:10:54.428809 30310 consensus_queue.cc:237] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } }
I20260923 18:10:54.430452 29692 catalog_manager.cc:5495] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0711054c596345308af6abe05a50d169 (127.27.75.193). New cstate: current_term: 1 leader_uuid: "0711054c596345308af6abe05a50d169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0711054c596345308af6abe05a50d169" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 } health_report { overall_health: UNKNOWN } } }
I20260923 18:10:54.453790 30325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:54.450738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.27.75.193:41513)
0923 18:10:54.451045 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:54.451052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:54.451062 (+ 10us) server_negotiation.cc:407] Connection header received
0923 18:10:54.451098 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:54.451100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:54.451138 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:54.451207 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:10:54.451435 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:54.451974 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:54.452771 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:54.453000 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:54.453085 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:54.453341 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:10:54.453418 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:10:54.453477 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:54.453566 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 18:10:54.453634 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":135,"threads_started":1}
I20260923 18:10:54.460215 30132 raft_consensus.cc:1261] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Refusing update from remote peer 0711054c596345308af6abe05a50d169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:54.460551 29993 raft_consensus.cc:1261] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Refusing update from remote peer 0711054c596345308af6abe05a50d169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:10:54.460565 30310 consensus_queue.cc:1035] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e492fe09301b4ae8962b2961838d2320" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:10:54.460875 30310 consensus_queue.cc:1035] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c944ff15a46461dae9a892b5b9d6276" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 18:10:54.476030 30275 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 18:10:56.026552 29690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:56.026700 29690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:10:56.028878 29690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dbed242d40b4242bb41603fc98b454a: Sending DeleteTablet for 3 replicas of tablet a612657fc6744bd885a35f1f52b6f4e2
I20260923 18:10:56.029386 29831 tablet_service.cc:1514] Processing DeleteTablet for tablet a612657fc6744bd885a35f1f52b6f4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:56 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:36306
I20260923 18:10:56.029497 29972 tablet_service.cc:1514] Processing DeleteTablet for tablet a612657fc6744bd885a35f1f52b6f4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:56 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:43324
I20260923 18:10:56.029778 30354 tablet_replica.cc:330] T a612657fc6744bd885a35f1f52b6f4e2 P 0711054c596345308af6abe05a50d169: stopping tablet replica
I20260923 18:10:56.029865 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 29766
I20260923 18:10:56.029996 30112 tablet_service.cc:1514] Processing DeleteTablet for tablet a612657fc6744bd885a35f1f52b6f4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:10:56 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:44172
I20260923 18:10:56.031356 30356 tablet_replica.cc:330] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320: stopping tablet replica
I20260923 18:10:56.031473 30356 raft_consensus.cc:2229] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:56.031596 30356 raft_consensus.cc:2258] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:56.031853 30355 tablet_replica.cc:330] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276: stopping tablet replica
I20260923 18:10:56.031978 30355 raft_consensus.cc:2229] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:10:56.032096 30355 raft_consensus.cc:2258] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:10:56.032467 30356 ts_tablet_manager.cc:1905] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:56.032673 30355 ts_tablet_manager.cc:1905] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:10:56.035161 30356 ts_tablet_manager.cc:1918] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 18:10:56.035161 30355 ts_tablet_manager.cc:1918] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 18:10:56.035243 30355 log.cc:1196] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/a612657fc6744bd885a35f1f52b6f4e2
I20260923 18:10:56.035248 30356 log.cc:1196] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/a612657fc6744bd885a35f1f52b6f4e2
I20260923 18:10:56.035490 30356 ts_tablet_manager.cc:1939] T a612657fc6744bd885a35f1f52b6f4e2 P e492fe09301b4ae8962b2961838d2320: Deleting consensus metadata
I20260923 18:10:56.035490 30355 ts_tablet_manager.cc:1939] T a612657fc6744bd885a35f1f52b6f4e2 P 9c944ff15a46461dae9a892b5b9d6276: Deleting consensus metadata
W20260923 18:10:56.036496 29679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:41513 (error 108)
I20260923 18:10:56.036590 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 29907
I20260923 18:10:56.036671 29680 catalog_manager.cc:4841] TS e492fe09301b4ae8962b2961838d2320 (127.27.75.195:35825): tablet a612657fc6744bd885a35f1f52b6f4e2 (table test-table [id=6a468e9a9a3b4da2ad6112add1c6da28]) successfully deleted
I20260923 18:10:56.042261 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:56.041766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:41513 (local address 127.0.0.1:36318)
0923 18:10:56.042032 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:56.042125 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:41513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
W20260923 18:10:56.042557 29679 catalog_manager.cc:4584] TS 0711054c596345308af6abe05a50d169 (127.27.75.193:41513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a612657fc6744bd885a35f1f52b6f4e2: Network error: Client connection negotiation failed: client connection to 127.27.75.193:41513: connect: Connection refused (error 111)
I20260923 18:10:56.042742 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 30048
I20260923 18:10:56.043524 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:56.042633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:38129 (local address 127.0.0.1:43352)
0923 18:10:56.042796 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:56.043439 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.194:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 18:10:56.048758 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 29661
2026-09-23T18:10:56Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6501 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:56 dist-test-slave-wsx5 krb5kdc[30361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:56 dist-test-slave-wsx5 krb5kdc[30361](info): set up 2 sockets
Sep 23 18:10:56 dist-test-slave-wsx5 krb5kdc[30361](info): commencing operation
krb5kdc: starting...
W20260923 18:10:57.003046 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:10:57 dist-test-slave-wsx5 krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:10:57.125344 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:40265
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:40265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:57.233387 30374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:57.233700 30374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:57.233750 30374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:57.237811 30374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:57.237879 30374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:57.237896 30374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:57.237911 30374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:57.237926 30374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:57.243393 30374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:40265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:40265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:57.244451 30374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:57.245527 30374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:57.254421 30382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:57.254424 30379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:57.254424 30380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:57.254529 30374 server_base.cc:1034] running on GCE node
I20260923 18:10:57.255323 30374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:57.256448 30374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:57.257681 30374 hybrid_clock.cc:648] HybridClock initialized: now 1790187057257664 us; error 36 us; skew 500 ppm
Sep 23 18:10:57 dist-test-slave-wsx5 krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:57.260515 30374 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:10:57.261524 30374 webserver.cc:460] Webserver started at http://127.27.75.254:43793/ using document root <none> and password file <none>
I20260923 18:10:57.262130 30374 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:57.262200 30374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:57.262394 30374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:57.264470 30374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "8e9bd5783d654363be71fded277cfa25"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "f6a3731684bff2665390478557052c22"
server_key_iv: "03d65fe21d2734e2d4f7ed8e20e25ad5"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.265010 30374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "8e9bd5783d654363be71fded277cfa25"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "f6a3731684bff2665390478557052c22"
server_key_iv: "03d65fe21d2734e2d4f7ed8e20e25ad5"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.268812 30374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 18:10:57.272315 30389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:57.273799 30374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 18:10:57.273926 30374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "8e9bd5783d654363be71fded277cfa25"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "f6a3731684bff2665390478557052c22"
server_key_iv: "03d65fe21d2734e2d4f7ed8e20e25ad5"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.274039 30374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:57.287176 30374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:57.290567 30374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:57.290766 30374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:57.297791 30374 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:40265
I20260923 18:10:57.297878 30441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:40265 every 8 connection(s)
I20260923 18:10:57.298856 30374 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:57.301268 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 30374
I20260923 18:10:57.301381 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:10:57.301632 27951 external_mini_cluster.cc:1428] Setting key dc89593cae95d84c79ba6daf7d2f0608
I20260923 18:10:57.302158 30442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:57.310029 30442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25: Bootstrap starting.
Sep 23 18:10:57 dist-test-slave-wsx5 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:57.312510 30442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:57.313434 30442 log.cc:824] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:57.315764 30442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25: No bootstrap required, opened a new log
I20260923 18:10:57.317799 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:57.303042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.27.75.254:40265)
0923 18:10:57.303505 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:57.303525 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:57.303569 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:10:57.304438 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:57.304466 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:57.304934 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:57.305400 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:57.306498 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:57.308079 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:57.308994 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:57.309726 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:57.309891 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:57.312342 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:57.312348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:57.312362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:57.312394 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:57.314195 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:57.314663 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:57.314668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:57.314675 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:57.314816 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:57.315160 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:57.315163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:57.315165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:57.315693 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:57.315873 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:57.316220 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 18:10:57.316532 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260923 18:10:57.319705 30442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9bd5783d654363be71fded277cfa25" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40265 } }
I20260923 18:10:57.319947 30442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:57.320007 30442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9bd5783d654363be71fded277cfa25, State: Initialized, Role: FOLLOWER
I20260923 18:10:57.320449 30442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9bd5783d654363be71fded277cfa25" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40265 } }
I20260923 18:10:57.320573 30442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:57.320616 30442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:57.320686 30442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:57.321661 30442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9bd5783d654363be71fded277cfa25" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40265 } }
I20260923 18:10:57.321964 30442 leader_election.cc:302] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e9bd5783d654363be71fded277cfa25; no voters:
I20260923 18:10:57.322315 30442 leader_election.cc:288] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:57.322415 30447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:57.322639 30447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [term 1 LEADER]: Becoming Leader. State: Replica: 8e9bd5783d654363be71fded277cfa25, State: Running, Role: LEADER
I20260923 18:10:57.322953 30447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9bd5783d654363be71fded277cfa25" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40265 } }
I20260923 18:10:57.323586 30442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:57.324558 30449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e9bd5783d654363be71fded277cfa25. Latest consensus state: current_term: 1 leader_uuid: "8e9bd5783d654363be71fded277cfa25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9bd5783d654363be71fded277cfa25" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40265 } } }
I20260923 18:10:57.324544 30448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e9bd5783d654363be71fded277cfa25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9bd5783d654363be71fded277cfa25" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40265 } } }
I20260923 18:10:57.324687 30449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:57.324687 30448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25 [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:57.325215 30456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:57.328363 30456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:57.334383 30456 catalog_manager.cc:1347] Generated new cluster ID: b01218ff3d774f4fb5306ab7ddf27911
I20260923 18:10:57.334477 30456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:57.344251 30456 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:57.345335 30456 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:57.350485 30456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e9bd5783d654363be71fded277cfa25: Generated new TSK 0
I20260923 18:10:57.351163 30456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:10:57.383657 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:40265
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:57.495990 30468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:57.496299 30468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:57.496343 30468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:57.500646 30468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:57.500730 30468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:57.500813 30468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:10:57.507897 30468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:40265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:57.509209 30468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:57.510257 30468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:57.519974 30473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:57.519974 30474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:57.520006 30476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:57.520309 30468 server_base.cc:1034] running on GCE node
I20260923 18:10:57.520727 30468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:57.521425 30468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:57.522599 30468 hybrid_clock.cc:648] HybridClock initialized: now 1790187057522561 us; error 50 us; skew 500 ppm
Sep 23 18:10:57 dist-test-slave-wsx5 krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:57.525444 30468 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:10:57.526326 30468 webserver.cc:460] Webserver started at http://127.27.75.193:37993/ using document root <none> and password file <none>
I20260923 18:10:57.526885 30468 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:57.526947 30468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:57.527143 30468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:57.529259 30468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "ad5cf5a695d846c691f3c3506bb1189a"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "8f75ef3351af6480cd208a9f352f94a6"
server_key_iv: "4fbc8047911aba21d173448f2b82f3ec"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.529912 30468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "ad5cf5a695d846c691f3c3506bb1189a"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "8f75ef3351af6480cd208a9f352f94a6"
server_key_iv: "4fbc8047911aba21d173448f2b82f3ec"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.533730 30468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:10:57.536314 30483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:57.537720 30468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:10:57.537834 30468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "ad5cf5a695d846c691f3c3506bb1189a"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "8f75ef3351af6480cd208a9f352f94a6"
server_key_iv: "4fbc8047911aba21d173448f2b82f3ec"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.537977 30468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:57.557392 30468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:57.560636 30468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:57.560819 30468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:57.561460 30468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:57.562299 30468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:57.562345 30468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:57.562387 30468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:57.562415 30468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:57.571669 30468 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:40583
I20260923 18:10:57.571774 30596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:40583 every 8 connection(s)
I20260923 18:10:57.572618 30468 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:10:57.580075 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 30468
I20260923 18:10:57.580183 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:10:57.580462 27951 external_mini_cluster.cc:1428] Setting key a55fc5197b854eaae70aa0b51f05be8c
Sep 23 18:10:57 dist-test-slave-wsx5 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:57.585963 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:57.574515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:40073 (local address 127.27.75.254:40265)
0923 18:10:57.574672 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:57.574676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:57.575696 (+ 1020us) server_negotiation.cc:407] Connection header received
0923 18:10:57.576563 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:57.576567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:57.576634 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:57.576703 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:57.577948 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:57.578561 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:57.579695 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:57.579948 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:57.580038 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:57.582615 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:57.582618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:57.582620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:57.582652 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:57.583919 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:57.584430 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:57.584433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:57.584434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:57.584512 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:57.584829 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:57.584832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:57.584834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:57.585015 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:57.585098 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:57.585656 (+ 558us) server_negotiation.cc:299] Negotiation successful
0923 18:10:57.585796 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:10:57.586833 30599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:57.574899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:40265 (local address 127.27.75.193:40073)
0923 18:10:57.575513 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:57.575553 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:57.576362 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:57.576880 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:57.576887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:57.577316 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:57.577805 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:57.577819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:57.578686 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:57.578688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:57.579577 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:57.579586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:57.580095 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:57.580099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:57.580303 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:57.580894 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:57.580911 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:57.582449 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:57.584033 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:57.584040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:57.584059 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:57.584335 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:57.584593 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:57.584596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:57.584598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:57.584743 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:57.585110 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:57.585147 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:57.585416 (+ 269us) client_negotiation.cc:769] Sending connection context
0923 18:10:57.585594 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 18:10:57.585946 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":96,"threads_started":1}
I20260923 18:10:57.588163 30597 heartbeater.cc:347] Connected to a master server at 127.27.75.254:40265
I20260923 18:10:57.588411 30597 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:10:57.588928 30597 heartbeater.cc:511] Master 127.27.75.254:40265 requested a full tablet report, sending...
I20260923 18:10:57.590636 30406 ts_manager.cc:194] Registered new tserver with Master: ad5cf5a695d846c691f3c3506bb1189a (127.27.75.193:40583)
I20260923 18:10:57.591842 30406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:40073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:10:57.609093 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:40265
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:57.721529 30602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:57.721827 30602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:57.721870 30602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:57.726651 30602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:57.726745 30602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:57.726835 30602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:10:57.734452 30602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:40265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:57.735553 30602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:57.736528 30602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:57.746382 30610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:57.746320 30607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:57.746454 30602 server_base.cc:1034] running on GCE node
W20260923 18:10:57.746564 30608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:57.747045 30602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:57.747759 30602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:57.748912 30602 hybrid_clock.cc:648] HybridClock initialized: now 1790187057748880 us; error 45 us; skew 500 ppm
Sep 23 18:10:57 dist-test-slave-wsx5 krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:57.751631 30602 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:10:57.752501 30602 webserver.cc:460] Webserver started at http://127.27.75.194:44061/ using document root <none> and password file <none>
I20260923 18:10:57.753242 30602 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:57.753332 30602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:57.753599 30602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:57.755779 30602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "23c0e49e5fe242589272aa4c3f07e604"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "638c63d1122588e18606995ce6589a8a"
server_key_iv: "dae435958a21c5300b03cf7a0b483f2f"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.756314 30602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "23c0e49e5fe242589272aa4c3f07e604"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "638c63d1122588e18606995ce6589a8a"
server_key_iv: "dae435958a21c5300b03cf7a0b483f2f"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.760250 30602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 18:10:57.762881 30617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:57.764145 30602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:10:57.764257 30602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "23c0e49e5fe242589272aa4c3f07e604"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "638c63d1122588e18606995ce6589a8a"
server_key_iv: "dae435958a21c5300b03cf7a0b483f2f"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.764374 30602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:57.780541 30602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:57.783722 30602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:57.783902 30602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:57.784502 30602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:57.785612 30602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:57.785662 30602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:57.785715 30602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:57.785745 30602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:57.794783 30602 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:39431
I20260923 18:10:57.794822 30730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:39431 every 8 connection(s)
I20260923 18:10:57.795699 30602 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
Sep 23 18:10:57 dist-test-slave-wsx5 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:57.805783 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 30602
I20260923 18:10:57.805912 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:10:57.806511 27951 external_mini_cluster.cc:1428] Setting key 49a649fb380fa2cbac2cb376cc72b0a0
I20260923 18:10:57.809594 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:57.797731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:39143 (local address 127.27.75.254:40265)
0923 18:10:57.797883 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:57.797886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:57.798871 (+ 985us) server_negotiation.cc:407] Connection header received
0923 18:10:57.799768 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:57.799771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:57.799820 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:57.799899 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:57.801101 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:57.801664 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:57.802894 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:57.803120 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:57.803206 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:57.805692 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:57.805697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:57.805699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:57.805732 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:57.807347 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:57.807951 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:57.807954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:57.807956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:57.808051 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:57.808412 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:57.808415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:57.808417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:57.808646 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:57.808739 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:57.809312 (+ 573us) server_negotiation.cc:299] Negotiation successful
0923 18:10:57.809441 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 18:10:57.810415 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:57.798142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:40265 (local address 127.27.75.194:39143)
0923 18:10:57.798749 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:57.798788 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:57.799526 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:57.800054 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:57.800061 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:57.800434 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:57.800957 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:57.800970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:57.801806 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:57.801809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:57.802761 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:57.802772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:57.803294 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:57.803297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:57.803452 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:57.804001 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:57.804018 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:57.805490 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:57.807465 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:57.807471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:57.807489 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:57.807849 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:57.808139 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:57.808143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:57.808144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:57.808318 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:57.808770 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:57.808775 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:57.809031 (+ 256us) client_negotiation.cc:769] Sending connection context
0923 18:10:57.809243 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 18:10:57.809525 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":99,"threads_started":1}
I20260923 18:10:57.811771 30731 heartbeater.cc:347] Connected to a master server at 127.27.75.254:40265
I20260923 18:10:57.812018 30731 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:57.812534 30731 heartbeater.cc:511] Master 127.27.75.254:40265 requested a full tablet report, sending...
I20260923 18:10:57.813676 30406 ts_manager.cc:194] Registered new tserver with Master: 23c0e49e5fe242589272aa4c3f07e604 (127.27.75.194:39431)
I20260923 18:10:57.814249 30406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:39143
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:10:57.836527 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:40265
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:57.947374 30736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:57.947651 30736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:57.947695 30736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:57.951814 30736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:57.951879 30736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:57.951939 30736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:10:57.957357 30736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:40265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:57.958431 30736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:57.959429 30736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:57.969470 30742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:57.969707 30741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:57.969494 30744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:57.969622 30736 server_base.cc:1034] running on GCE node
I20260923 18:10:57.970247 30736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:57.970944 30736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:57.972155 30736 hybrid_clock.cc:648] HybridClock initialized: now 1790187057972110 us; error 53 us; skew 500 ppm
Sep 23 18:10:57 dist-test-slave-wsx5 krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:57.975104 30736 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:10:57.976085 30736 webserver.cc:460] Webserver started at http://127.27.75.195:41559/ using document root <none> and password file <none>
I20260923 18:10:57.976630 30736 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:57.976691 30736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:57.976859 30736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:57.979045 30736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "63f949033ec14922a366110c56c45518"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "7c27fbc29ba12e916405f8eef1f127c7"
server_key_iv: "02a8c6ca52c01a5a8b599229f7b05a8c"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.979578 30736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "63f949033ec14922a366110c56c45518"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "7c27fbc29ba12e916405f8eef1f127c7"
server_key_iv: "02a8c6ca52c01a5a8b599229f7b05a8c"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.983412 30736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:10:57.985988 30751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:57.987267 30736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:10:57.987376 30736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "63f949033ec14922a366110c56c45518"
format_stamp: "Formatted at 2026-09-23 18:10:57 on dist-test-slave-wsx5"
server_key: "7c27fbc29ba12e916405f8eef1f127c7"
server_key_iv: "02a8c6ca52c01a5a8b599229f7b05a8c"
server_key_version: "encryptionkey@0"
I20260923 18:10:57.987480 30736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:58.001555 30736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:58.004932 30736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:58.005153 30736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:58.005801 30736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:10:58.006626 30736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:10:58.006676 30736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:58.006729 30736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:10:58.006757 30736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:58.014900 30736 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:44135
I20260923 18:10:58.014932 30864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:44135 every 8 connection(s)
I20260923 18:10:58.015758 30736 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:10:58.023061 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 30736
I20260923 18:10:58.023176 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:10:58.023451 27951 external_mini_cluster.cc:1428] Setting key 560dd1e8b18b04bb4e2fd2c4dbdb0ded
Sep 23 18:10:58 dist-test-slave-wsx5 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:58.029253 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:58.017530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:60767 (local address 127.27.75.254:40265)
0923 18:10:58.017667 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:58.017670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:58.018451 (+ 781us) server_negotiation.cc:407] Connection header received
0923 18:10:58.019695 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:58.019698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:58.019746 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:58.019819 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:58.020971 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:58.021571 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:58.022729 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:58.022981 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:58.023090 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:58.025870 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:58.025874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:58.025877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:58.025907 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:58.027270 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:58.027773 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:58.027776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:58.027777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:58.027869 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:58.028160 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:58.028162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:58.028163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:58.028361 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:58.028440 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:58.028961 (+ 521us) server_negotiation.cc:299] Negotiation successful
0923 18:10:58.029071 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:10:58.030072 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:58.017881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:40265 (local address 127.27.75.195:60767)
0923 18:10:58.018299 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0923 18:10:58.018346 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:10:58.019447 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:10:58.019965 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:10:58.019972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:10:58.020306 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:10:58.020820 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:58.020834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:58.021685 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:58.021688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:58.022597 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:10:58.022607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:10:58.023164 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:10:58.023168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:10:58.023380 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:58.023949 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:10:58.023965 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:10:58.025690 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:10:58.027377 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:58.027382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:58.027396 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:58.027654 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:58.027947 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:10:58.027949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:10:58.027951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:10:58.028086 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:10:58.028454 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:10:58.028460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:10:58.028716 (+ 256us) client_negotiation.cc:769] Sending connection context
0923 18:10:58.028916 (+ 200us) client_negotiation.cc:240] Negotiation successful
0923 18:10:58.029236 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260923 18:10:58.031198 30865 heartbeater.cc:347] Connected to a master server at 127.27.75.254:40265
I20260923 18:10:58.031432 30865 heartbeater.cc:464] Registering TS with master...
I20260923 18:10:58.031957 30865 heartbeater.cc:511] Master 127.27.75.254:40265 requested a full tablet report, sending...
I20260923 18:10:58.033054 30406 ts_manager.cc:194] Registered new tserver with Master: 63f949033ec14922a366110c56c45518 (127.27.75.195:44135)
I20260923 18:10:58.033744 30406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:60767
I20260923 18:10:58.038050 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 18:10:58.050390 30445 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:10:58.048933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.27.75.254:40265)
0923 18:10:58.049080 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:58.049085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:58.049237 (+ 152us) server_negotiation.cc:407] Connection header received
0923 18:10:58.049393 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:58.049396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:58.049441 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:58.049504 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:58.050280 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54806: BlockingRecv error: recv got EOF from 127.0.0.1:54806 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:10:58.051527 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 30468
I20260923 18:10:58.056290 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 30602
I20260923 18:10:58.060719 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 30736
I20260923 18:10:58.065330 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 30374
2026-09-23T18:10:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2014 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:58 dist-test-slave-wsx5 krb5kdc[30877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:58 dist-test-slave-wsx5 krb5kdc[30877](info): set up 2 sockets
Sep 23 18:10:58 dist-test-slave-wsx5 krb5kdc[30877](info): commencing operation
krb5kdc: starting...
W20260923 18:10:59.023098 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:10:59 dist-test-slave-wsx5 krb5kdc[30877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:10:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:10:59.144402 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38343
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:38343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:10:59.254390 30890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:10:59.254680 30890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:10:59.254740 30890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:10:59.258845 30890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:10:59.258914 30890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:10:59.258941 30890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:10:59.258965 30890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:10:59.258983 30890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:10:59.264350 30890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:38343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:10:59.265489 30890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:10:59.266551 30890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:10:59.275156 30898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:59.275254 30896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:10:59.275151 30895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:10:59.275408 30890 server_base.cc:1034] running on GCE node
I20260923 18:10:59.276173 30890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:10:59.277616 30890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:10:59.278813 30890 hybrid_clock.cc:648] HybridClock initialized: now 1790187059278790 us; error 43 us; skew 500 ppm
Sep 23 18:10:59 dist-test-slave-wsx5 krb5kdc[30877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187059, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:10:59.287405 30890 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:10:59.288465 30890 webserver.cc:460] Webserver started at http://127.27.75.254:41753/ using document root <none> and password file <none>
I20260923 18:10:59.289028 30890 fs_manager.cc:359] Metadata directory not provided
I20260923 18:10:59.289095 30890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:10:59.289331 30890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:10:59.291460 30890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "500bee349cb64ab5ad3e74a98de2aa8f"
format_stamp: "Formatted at 2026-09-23 18:10:59 on dist-test-slave-wsx5"
server_key: "1c98b5689e0a1a3f124e08049f947a3b"
server_key_iv: "c2f32b7e925647d350a17895467d671c"
server_key_version: "encryptionkey@0"
I20260923 18:10:59.292016 30890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "500bee349cb64ab5ad3e74a98de2aa8f"
format_stamp: "Formatted at 2026-09-23 18:10:59 on dist-test-slave-wsx5"
server_key: "1c98b5689e0a1a3f124e08049f947a3b"
server_key_iv: "c2f32b7e925647d350a17895467d671c"
server_key_version: "encryptionkey@0"
I20260923 18:10:59.295939 30890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:10:59.298614 30905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:10:59.299932 30890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:10:59.300055 30890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "500bee349cb64ab5ad3e74a98de2aa8f"
format_stamp: "Formatted at 2026-09-23 18:10:59 on dist-test-slave-wsx5"
server_key: "1c98b5689e0a1a3f124e08049f947a3b"
server_key_iv: "c2f32b7e925647d350a17895467d671c"
server_key_version: "encryptionkey@0"
I20260923 18:10:59.300180 30890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:10:59.339710 30890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:10:59.342852 30890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:10:59.343029 30890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:10:59.349710 30890 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:38343
I20260923 18:10:59.349963 30957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:38343 every 8 connection(s)
I20260923 18:10:59.350759 30890 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:10:59.351301 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 30890
I20260923 18:10:59.351426 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:10:59.351673 27951 external_mini_cluster.cc:1428] Setting key 36b29f42b42030153864222eb5be5011
I20260923 18:10:59.354101 30958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:10:59.360884 30958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f: Bootstrap starting.
Sep 23 18:10:59 dist-test-slave-wsx5 krb5kdc[30877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:10:59.363794 30958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f: Neither blocks nor log segments found. Creating new log.
I20260923 18:10:59.364703 30958 log.cc:824] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:10:59.366972 30958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f: No bootstrap required, opened a new log
I20260923 18:10:59.367300 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:59.353072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.27.75.254:38343)
0923 18:10:59.353541 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:59.353559 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:59.353617 (+ 58us) server_negotiation.cc:407] Connection header received
0923 18:10:59.354582 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:59.354605 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:59.355018 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:59.355394 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:59.356084 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:59.357638 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:59.358590 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:59.359276 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:59.359436 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:59.362109 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:59.362118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:59.362145 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:10:59.362195 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:59.364240 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:59.364725 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:59.364732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:59.364738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:59.364881 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:10:59.365205 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:10:59.365208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:10:59.365210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:10:59.365636 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:59.365837 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:59.366124 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 18:10:59.366434 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":121,"threads_started":1}
I20260923 18:10:59.370252 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:10:59.370800 30958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bee349cb64ab5ad3e74a98de2aa8f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38343 } }
I20260923 18:10:59.371042 30958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:10:59.371101 30958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500bee349cb64ab5ad3e74a98de2aa8f, State: Initialized, Role: FOLLOWER
I20260923 18:10:59.371516 30958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bee349cb64ab5ad3e74a98de2aa8f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38343 } }
I20260923 18:10:59.371637 30958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:10:59.371696 30958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:10:59.371788 30958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:10:59.372711 30958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bee349cb64ab5ad3e74a98de2aa8f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38343 } }
I20260923 18:10:59.373054 30958 leader_election.cc:302] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500bee349cb64ab5ad3e74a98de2aa8f; no voters:
I20260923 18:10:59.373456 30958 leader_election.cc:288] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:10:59.373656 30964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:10:59.373940 30964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [term 1 LEADER]: Becoming Leader. State: Replica: 500bee349cb64ab5ad3e74a98de2aa8f, State: Running, Role: LEADER
I20260923 18:10:59.374315 30964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bee349cb64ab5ad3e74a98de2aa8f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38343 } }
I20260923 18:10:59.374886 30958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:10:59.375914 30966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500bee349cb64ab5ad3e74a98de2aa8f. Latest consensus state: current_term: 1 leader_uuid: "500bee349cb64ab5ad3e74a98de2aa8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bee349cb64ab5ad3e74a98de2aa8f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38343 } } }
I20260923 18:10:59.376039 30966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:59.376355 30965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500bee349cb64ab5ad3e74a98de2aa8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500bee349cb64ab5ad3e74a98de2aa8f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38343 } } }
I20260923 18:10:59.376437 30965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f [sys.catalog]: This master's current role is: LEADER
I20260923 18:10:59.376870 30973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:10:59.380735 30973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:10:59.387042 30973 catalog_manager.cc:1347] Generated new cluster ID: 6a9d43b9a8724311adaa716f214620a8
I20260923 18:10:59.387115 30973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:10:59.388887 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:10:59.383301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.27.75.254:38343)
0923 18:10:59.383413 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 18:10:59.383417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:10:59.383742 (+ 325us) server_negotiation.cc:407] Connection header received
0923 18:10:59.383877 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:10:59.383880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:10:59.383932 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:10:59.384031 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:10:59.384675 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:59.385318 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:59.386387 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:10:59.386781 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:10:59.386887 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:10:59.387937 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:10:59.387940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:10:59.387943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:10:59.388438 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:10:59.388471 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:10:59.388489 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:10:59.388568 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 18:10:59.388684 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 18:10:59.388749 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 18:10:59.393697 30973 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:10:59.394570 30973 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:10:59.402463 30973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500bee349cb64ab5ad3e74a98de2aa8f: Generated new TSK 0
I20260923 18:10:59.403242 30973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:10:59.410199 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 30890
2026-09-23T18:10:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1343 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:10:59 dist-test-slave-wsx5 krb5kdc[30991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:10:59 dist-test-slave-wsx5 krb5kdc[30991](info): set up 2 sockets
Sep 23 18:10:59 dist-test-slave-wsx5 krb5kdc[30991](info): commencing operation
krb5kdc: starting...
W20260923 18:11:00.375135 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:11:00 dist-test-slave-wsx5 krb5kdc[30991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:11:00.498644 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38715
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:38715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:00.617625 31004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:00.617961 31004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:00.618003 31004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:00.622442 31004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:00.622531 31004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:00.622551 31004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:00.622567 31004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:00.622582 31004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:00.628491 31004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:38715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:00.629709 31004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:00.630800 31004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:00.640507 31012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:00.640501 31009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:00.640579 31004 server_base.cc:1034] running on GCE node
W20260923 18:11:00.640501 31010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:00.641397 31004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:00.642606 31004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:00.643787 31004 hybrid_clock.cc:648] HybridClock initialized: now 1790187060643755 us; error 44 us; skew 500 ppm
Sep 23 18:11:00 dist-test-slave-wsx5 krb5kdc[30991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187060, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:00.646775 31004 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:11:00.647688 31004 webserver.cc:460] Webserver started at http://127.27.75.254:44073/ using document root <none> and password file <none>
I20260923 18:11:00.648227 31004 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:00.648288 31004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:00.648451 31004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:00.650766 31004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "dc980dea52c74712835fab84a6de854f"
format_stamp: "Formatted at 2026-09-23 18:11:00 on dist-test-slave-wsx5"
server_key: "e8dab1cc2b4519dc75f763b54ea8248e"
server_key_iv: "d1568c341d0980b26fd8e7f475a17cd3"
server_key_version: "encryptionkey@0"
I20260923 18:11:00.651352 31004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "dc980dea52c74712835fab84a6de854f"
format_stamp: "Formatted at 2026-09-23 18:11:00 on dist-test-slave-wsx5"
server_key: "e8dab1cc2b4519dc75f763b54ea8248e"
server_key_iv: "d1568c341d0980b26fd8e7f475a17cd3"
server_key_version: "encryptionkey@0"
I20260923 18:11:00.655437 31004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:11:00.658003 31019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:00.659368 31004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:00.659500 31004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "dc980dea52c74712835fab84a6de854f"
format_stamp: "Formatted at 2026-09-23 18:11:00 on dist-test-slave-wsx5"
server_key: "e8dab1cc2b4519dc75f763b54ea8248e"
server_key_iv: "d1568c341d0980b26fd8e7f475a17cd3"
server_key_version: "encryptionkey@0"
I20260923 18:11:00.659612 31004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:00.683295 31004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:00.686836 31004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:00.687043 31004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:00.694083 31004 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:38715
I20260923 18:11:00.694344 31071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:38715 every 8 connection(s)
I20260923 18:11:00.695134 31004 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:00.695595 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 31004
I20260923 18:11:00.695725 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:00.696022 27951 external_mini_cluster.cc:1428] Setting key c2f09be6016f33f65fdd499f64820ea4
I20260923 18:11:00.698619 31072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:00.705549 31072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f: Bootstrap starting.
Sep 23 18:11:00 dist-test-slave-wsx5 krb5kdc[30991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:00.708218 31072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:00.709060 31072 log.cc:824] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:00.711176 31072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f: No bootstrap required, opened a new log
I20260923 18:11:00.711550 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:00.697587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.27.75.254:38715)
0923 18:11:00.698027 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:00.698046 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:00.698090 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:11:00.698948 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:00.698973 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:00.699411 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:00.699788 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:00.700522 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:00.702088 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:00.703040 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:00.703745 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:00.703903 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:00.706813 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:00.706819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:00.706832 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:00.706867 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:00.708573 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:00.709089 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:00.709094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:00.709099 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:00.709283 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:00.709576 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:00.709580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:00.709582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:00.710001 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:00.710187 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:00.710494 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 18:11:00.710790 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20260923 18:11:00.714867 31072 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc980dea52c74712835fab84a6de854f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38715 } }
I20260923 18:11:00.715123 31072 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:00.715163 31072 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc980dea52c74712835fab84a6de854f, State: Initialized, Role: FOLLOWER
I20260923 18:11:00.715571 31072 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc980dea52c74712835fab84a6de854f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38715 } }
I20260923 18:11:00.715698 31072 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:00.715737 31072 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:00.715813 31072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:00.716750 31072 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc980dea52c74712835fab84a6de854f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38715 } }
I20260923 18:11:00.717085 31072 leader_election.cc:302] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc980dea52c74712835fab84a6de854f; no voters:
I20260923 18:11:00.717443 31072 leader_election.cc:288] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:00.717602 31077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:00.717892 31077 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [term 1 LEADER]: Becoming Leader. State: Replica: dc980dea52c74712835fab84a6de854f, State: Running, Role: LEADER
I20260923 18:11:00.718212 31077 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc980dea52c74712835fab84a6de854f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38715 } }
I20260923 18:11:00.718609 31072 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:00.719909 31079 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc980dea52c74712835fab84a6de854f. Latest consensus state: current_term: 1 leader_uuid: "dc980dea52c74712835fab84a6de854f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc980dea52c74712835fab84a6de854f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38715 } } }
I20260923 18:11:00.719954 31078 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc980dea52c74712835fab84a6de854f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc980dea52c74712835fab84a6de854f" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38715 } } }
I20260923 18:11:00.720067 31078 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:00.720017 31079 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f [sys.catalog]: This master's current role is: LEADER
W20260923 18:11:00.723014 31092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:11:00.723088 31092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:11:00.723163 31088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:00.724025 31088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:00.730000 31088 catalog_manager.cc:1347] Generated new cluster ID: 3b49d2b1956a4b16b4650961d3cb3dc0
I20260923 18:11:00.730096 31088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:00.745512 31088 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:00.746601 31088 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:00.754624 31088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc980dea52c74712835fab84a6de854f: Generated new TSK 0
I20260923 18:11:00.755347 31088 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:11:00.790519 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38715
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:00.904379 31098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:00.904675 31098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:00.904719 31098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:00.908901 31098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:00.908983 31098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:00.909047 31098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:00.914460 31098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:00.915520 31098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:00.916589 31098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:00.926559 31104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:00.926599 31098 server_base.cc:1034] running on GCE node
W20260923 18:11:00.926564 31103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:00.926559 31106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:00.927230 31098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:00.927858 31098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:00.929020 31098 hybrid_clock.cc:648] HybridClock initialized: now 1790187060929000 us; error 33 us; skew 500 ppm
Sep 23 18:11:00 dist-test-slave-wsx5 krb5kdc[30991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187060, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:00.931886 31098 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:11:00.932878 31098 webserver.cc:460] Webserver started at http://127.27.75.193:35679/ using document root <none> and password file <none>
I20260923 18:11:00.933496 31098 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:00.933589 31098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:00.933815 31098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:00.935988 31098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "bfae4d0db21b4c7da5e4741bec1fce85"
format_stamp: "Formatted at 2026-09-23 18:11:00 on dist-test-slave-wsx5"
server_key: "b26603946bea1c2872883d6e9b68a43c"
server_key_iv: "b07bc4de082f6ac3dc1fbbd191740135"
server_key_version: "encryptionkey@0"
I20260923 18:11:00.936570 31098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "bfae4d0db21b4c7da5e4741bec1fce85"
format_stamp: "Formatted at 2026-09-23 18:11:00 on dist-test-slave-wsx5"
server_key: "b26603946bea1c2872883d6e9b68a43c"
server_key_iv: "b07bc4de082f6ac3dc1fbbd191740135"
server_key_version: "encryptionkey@0"
I20260923 18:11:00.940428 31098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:11:00.943085 31113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:00.944449 31098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:11:00.944561 31098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "bfae4d0db21b4c7da5e4741bec1fce85"
format_stamp: "Formatted at 2026-09-23 18:11:00 on dist-test-slave-wsx5"
server_key: "b26603946bea1c2872883d6e9b68a43c"
server_key_iv: "b07bc4de082f6ac3dc1fbbd191740135"
server_key_version: "encryptionkey@0"
I20260923 18:11:00.944658 31098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:00.955905 31098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:00.959231 31098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:00.959408 31098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:00.960011 31098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:00.960829 31098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:00.960875 31098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:00.960927 31098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:00.960953 31098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:00.970067 31098 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:35125
I20260923 18:11:00.970201 31226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:35125 every 8 connection(s)
I20260923 18:11:00.971033 31098 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:00.977044 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 31098
I20260923 18:11:00.977205 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:11:00.977573 27951 external_mini_cluster.cc:1428] Setting key 984c29be41c0360258a21744b1428e16
Sep 23 18:11:00 dist-test-slave-wsx5 krb5kdc[30991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187060, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:11:00.986759 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:00.973017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:49549 (local address 127.27.75.254:38715)
0923 18:11:00.973235 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:00.973240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:00.974259 (+ 1019us) server_negotiation.cc:407] Connection header received
0923 18:11:00.975800 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:00.975804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:00.975869 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:00.975945 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:00.977335 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:00.978111 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:00.979499 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:00.979838 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:00.979956 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:00.982895 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:00.982898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:00.982900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:00.982930 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:00.984550 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:00.985187 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:00.985190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:00.985191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:00.985284 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:00.985630 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:00.985633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:00.985634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:00.985831 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:00.985918 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:00.986442 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 18:11:00.986581 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 18:11:00.987622 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:00.973504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38715 (local address 127.27.75.193:49549)
0923 18:11:00.974030 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:00.974076 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:00.975533 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:00.976152 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:00.976160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:00.976551 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:00.977110 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:00.977160 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:00.978262 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:00.978265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:00.979341 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:00.979352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:00.980031 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:00.980034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:00.980239 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:00.980953 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:00.980972 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:00.982684 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:00.984695 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:00.984703 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:00.984724 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:00.985050 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:00.985377 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:00.985381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:00.985382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:00.985532 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:00.985931 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:00.985939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:00.986201 (+ 262us) client_negotiation.cc:769] Sending connection context
0923 18:11:00.986371 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 18:11:00.986725 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":139,"threads_started":1}
I20260923 18:11:00.989159 31227 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38715
I20260923 18:11:00.989526 31227 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:11:00.990196 31227 heartbeater.cc:511] Master 127.27.75.254:38715 requested a full tablet report, sending...
I20260923 18:11:00.992025 31036 ts_manager.cc:194] Registered new tserver with Master: bfae4d0db21b4c7da5e4741bec1fce85 (127.27.75.193:35125)
I20260923 18:11:00.993347 31036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:49549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:11:01.008674 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38715
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:01.121515 31232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:01.121810 31232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:01.121871 31232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:01.126003 31232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:01.126076 31232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:01.126163 31232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:01.131649 31232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:01.132784 31232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:01.133908 31232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:11:01.144308 31232 server_base.cc:1034] running on GCE node
W20260923 18:11:01.144219 31237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:01.144326 31238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:01.144220 31240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:01.144953 31232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:01.145813 31232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:01.146999 31232 hybrid_clock.cc:648] HybridClock initialized: now 1790187061146969 us; error 55 us; skew 500 ppm
Sep 23 18:11:01 dist-test-slave-wsx5 krb5kdc[30991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187061, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:01.150195 31232 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:11:01.151364 31232 webserver.cc:460] Webserver started at http://127.27.75.194:36395/ using document root <none> and password file <none>
I20260923 18:11:01.152031 31232 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:01.152109 31232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:01.152325 31232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:01.154536 31232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "6954b3ffe21246ce844e9e2f031cb6fb"
format_stamp: "Formatted at 2026-09-23 18:11:01 on dist-test-slave-wsx5"
server_key: "067b6f78ac1c585e583d1c8b1dd2a462"
server_key_iv: "3204595cc5cbbd0860205bd2809ea3b0"
server_key_version: "encryptionkey@0"
I20260923 18:11:01.155122 31232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "6954b3ffe21246ce844e9e2f031cb6fb"
format_stamp: "Formatted at 2026-09-23 18:11:01 on dist-test-slave-wsx5"
server_key: "067b6f78ac1c585e583d1c8b1dd2a462"
server_key_iv: "3204595cc5cbbd0860205bd2809ea3b0"
server_key_version: "encryptionkey@0"
I20260923 18:11:01.159101 31232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260923 18:11:01.161876 31247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:01.163444 31232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 18:11:01.163573 31232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "6954b3ffe21246ce844e9e2f031cb6fb"
format_stamp: "Formatted at 2026-09-23 18:11:01 on dist-test-slave-wsx5"
server_key: "067b6f78ac1c585e583d1c8b1dd2a462"
server_key_iv: "3204595cc5cbbd0860205bd2809ea3b0"
server_key_version: "encryptionkey@0"
I20260923 18:11:01.163691 31232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:01.176730 31232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:01.180055 31232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:01.180258 31232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:01.180850 31232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:01.181730 31232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:01.181778 31232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:01.181834 31232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:01.181864 31232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:01.190963 31232 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:40341
I20260923 18:11:01.191015 31360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:40341 every 8 connection(s)
I20260923 18:11:01.191941 31232 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:01.195559 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 31232
I20260923 18:11:01.195676 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:11:01.195973 27951 external_mini_cluster.cc:1428] Setting key 2c51455286367274721736a137f88e48
Sep 23 18:11:01 dist-test-slave-wsx5 krb5kdc[30991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187061, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:11:01.208052 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:01.193930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:59021 (local address 127.27.75.254:38715)
0923 18:11:01.194079 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:01.194084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:01.195033 (+ 949us) server_negotiation.cc:407] Connection header received
0923 18:11:01.196489 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:01.196493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:01.196551 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:01.196633 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:01.198159 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:01.198895 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:01.200781 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:01.201046 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:01.201212 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:01.204439 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:01.204443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:01.204446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:01.204476 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:01.206014 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:01.206537 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:01.206539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:01.206541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:01.206614 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:01.206936 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:01.206939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:01.206940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:01.207126 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:01.207201 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:01.207802 (+ 601us) server_negotiation.cc:299] Negotiation successful
0923 18:11:01.207915 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
I20260923 18:11:01.208984 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:01.194363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38715 (local address 127.27.75.194:59021)
0923 18:11:01.194872 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:01.194920 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:01.196230 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:01.196836 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:01.196844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:01.197466 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:01.197993 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:01.198007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:01.199070 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:01.199078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:01.200567 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:01.200580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:01.201265 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:01.201269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:01.201495 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:01.202154 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:01.202188 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:01.204124 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:01.206133 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:01.206139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:01.206154 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:01.206452 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:01.206703 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:01.206706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:01.206708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:01.206856 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:01.207231 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:01.207236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:01.207554 (+ 318us) client_negotiation.cc:769] Sending connection context
0923 18:11:01.207728 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 18:11:01.208057 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":129,"threads_started":1}
I20260923 18:11:01.210335 31361 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38715
I20260923 18:11:01.210593 31361 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:01.211109 31361 heartbeater.cc:511] Master 127.27.75.254:38715 requested a full tablet report, sending...
I20260923 18:11:01.212239 31036 ts_manager.cc:194] Registered new tserver with Master: 6954b3ffe21246ce844e9e2f031cb6fb (127.27.75.194:40341)
I20260923 18:11:01.213025 31036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:59021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:11:01.225188 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38715
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:01.340169 31366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:01.340529 31366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:01.340591 31366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:01.344739 31366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:01.344818 31366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:01.344878 31366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:01.350394 31366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:01.351432 31366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:01.352471 31366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:01.362563 31371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:01.362565 31374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:01.362601 31366 server_base.cc:1034] running on GCE node
W20260923 18:11:01.362900 31372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:01.363338 31366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:01.364061 31366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:01.365211 31366 hybrid_clock.cc:648] HybridClock initialized: now 1790187061365183 us; error 37 us; skew 500 ppm
Sep 23 18:11:01 dist-test-slave-wsx5 krb5kdc[30991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187061, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:01.368201 31366 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:11:01.369517 31366 webserver.cc:460] Webserver started at http://127.27.75.195:33027/ using document root <none> and password file <none>
I20260923 18:11:01.370146 31366 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:01.370214 31366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:01.370383 31366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:01.372550 31366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "f2758a6f1a8045db8649f54ef4f7442e"
format_stamp: "Formatted at 2026-09-23 18:11:01 on dist-test-slave-wsx5"
server_key: "2341b4b58ad52bba759e057482a1d80e"
server_key_iv: "77227760a41a166d52ccfd2b32ac23e7"
server_key_version: "encryptionkey@0"
I20260923 18:11:01.373168 31366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "f2758a6f1a8045db8649f54ef4f7442e"
format_stamp: "Formatted at 2026-09-23 18:11:01 on dist-test-slave-wsx5"
server_key: "2341b4b58ad52bba759e057482a1d80e"
server_key_iv: "77227760a41a166d52ccfd2b32ac23e7"
server_key_version: "encryptionkey@0"
I20260923 18:11:01.377207 31366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 18:11:01.379912 31381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:01.381349 31366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 18:11:01.381474 31366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "f2758a6f1a8045db8649f54ef4f7442e"
format_stamp: "Formatted at 2026-09-23 18:11:01 on dist-test-slave-wsx5"
server_key: "2341b4b58ad52bba759e057482a1d80e"
server_key_iv: "77227760a41a166d52ccfd2b32ac23e7"
server_key_version: "encryptionkey@0"
I20260923 18:11:01.381599 31366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:01.405503 31366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:01.408644 31366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:01.408825 31366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:01.409478 31366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:01.410328 31366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:01.410377 31366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:01.410431 31366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:01.410458 31366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:01.419464 31366 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:40809
I20260923 18:11:01.419485 31494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:40809 every 8 connection(s)
I20260923 18:11:01.420408 31366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:01.421761 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 31366
I20260923 18:11:01.421875 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:11:01.422130 27951 external_mini_cluster.cc:1428] Setting key 096b9e9fa0ff01905fb42f5ea88bf224
Sep 23 18:11:01 dist-test-slave-wsx5 krb5kdc[30991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187061, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:01.434207 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:01.422347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:60129 (local address 127.27.75.254:38715)
0923 18:11:01.422483 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:01.422487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:01.423449 (+ 962us) server_negotiation.cc:407] Connection header received
0923 18:11:01.424917 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:01.424920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:01.424971 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:01.425042 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:01.426215 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:01.426735 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:01.427839 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:01.428052 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:01.428136 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:01.430564 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:01.430566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:01.430568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:01.430595 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:01.432041 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:01.432582 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:01.432585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:01.432586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:01.432670 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:01.433002 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:01.433006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:01.433007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:01.433259 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:01.433349 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:01.433897 (+ 548us) server_negotiation.cc:299] Negotiation successful
0923 18:11:01.434036 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:11:01.435214 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:01.422780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38715 (local address 127.27.75.195:60129)
0923 18:11:01.423312 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:01.423366 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:01.424726 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:01.425250 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:01.425258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:01.425607 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:01.426082 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:01.426094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:01.426847 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:01.426851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:01.427729 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:01.427738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:01.428193 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:01.428197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:01.428351 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:01.428908 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:01.428924 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:01.430392 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:01.432175 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:01.432182 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:01.432199 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:01.432481 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:01.432759 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:01.432762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:01.432764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:01.432909 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:01.433358 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:01.433364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:01.433651 (+ 287us) client_negotiation.cc:769] Sending connection context
0923 18:11:01.433840 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 18:11:01.434192 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":143,"threads_started":1}
I20260923 18:11:01.436429 31495 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38715
I20260923 18:11:01.436642 31495 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:01.437096 31495 heartbeater.cc:511] Master 127.27.75.254:38715 requested a full tablet report, sending...
I20260923 18:11:01.438252 31035 ts_manager.cc:194] Registered new tserver with Master: f2758a6f1a8045db8649f54ef4f7442e (127.27.75.195:40809)
I20260923 18:11:01.438840 31035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:60129
I20260923 18:11:01.448491 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 18:11:01 dist-test-slave-wsx5 krb5kdc[30991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187061, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 18:11:01 dist-test-slave-wsx5 krb5kdc[30991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187061, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:01.474794 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:01.465298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.27.75.254:38715)
0923 18:11:01.465464 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:01.465468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:01.465706 (+ 238us) server_negotiation.cc:407] Connection header received
0923 18:11:01.465839 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:01.465842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:01.465887 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:01.465970 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:01.466831 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:01.467400 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:01.468355 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:01.468591 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:01.468727 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:01.471233 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:01.471237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:01.471239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:01.471270 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:01.472719 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:01.473253 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:01.473257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:01.473258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:01.473349 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:01.473751 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:01.473754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:01.473756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:01.474022 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:01.474134 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:01.474434 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 18:11:01.474571 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 18:11:01.475271 31035 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46236
I20260923 18:11:01.476205 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 31098
I20260923 18:11:01.481441 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 31232
I20260923 18:11:01.486266 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 31366
I20260923 18:11:01.491297 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 31004
2026-09-23T18:11:01Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2083 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:11:01 dist-test-slave-wsx5 krb5kdc[31507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:11:01 dist-test-slave-wsx5 krb5kdc[31507](info): set up 2 sockets
Sep 23 18:11:01 dist-test-slave-wsx5 krb5kdc[31507](info): commencing operation
krb5kdc: starting...
W20260923 18:11:02.459157 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:11:02 dist-test-slave-wsx5 krb5kdc[31507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:11:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:11:02.588578 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:41757
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:41757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:02.704583 31520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:02.704898 31520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:02.704941 31520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:02.709187 31520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:02.709262 31520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:02.709280 31520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:02.709296 31520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:02.709311 31520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:02.714938 31520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:41757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:41757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:02.716012 31520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:02.717191 31520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:02.725785 31525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:02.725822 31526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:02.725837 31520 server_base.cc:1034] running on GCE node
W20260923 18:11:02.725785 31528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:02.726650 31520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:02.727813 31520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:02.729008 31520 hybrid_clock.cc:648] HybridClock initialized: now 1790187062728984 us; error 38 us; skew 500 ppm
Sep 23 18:11:02 dist-test-slave-wsx5 krb5kdc[31507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187062, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:02.732035 31520 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:11:02.733279 31520 webserver.cc:460] Webserver started at http://127.27.75.254:38703/ using document root <none> and password file <none>
I20260923 18:11:02.733899 31520 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:02.733968 31520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:02.734163 31520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:02.736266 31520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "fa232b075d1242238e82df40df8b66d9"
format_stamp: "Formatted at 2026-09-23 18:11:02 on dist-test-slave-wsx5"
server_key: "8a6cdc830a2f039e566bd002e19a54f4"
server_key_iv: "888d4b4adc442e29d7f0b13717f3aa28"
server_key_version: "encryptionkey@0"
I20260923 18:11:02.736835 31520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "fa232b075d1242238e82df40df8b66d9"
format_stamp: "Formatted at 2026-09-23 18:11:02 on dist-test-slave-wsx5"
server_key: "8a6cdc830a2f039e566bd002e19a54f4"
server_key_iv: "888d4b4adc442e29d7f0b13717f3aa28"
server_key_version: "encryptionkey@0"
I20260923 18:11:02.741074 31520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 18:11:02.744340 31535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:02.745819 31520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:02.745944 31520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "fa232b075d1242238e82df40df8b66d9"
format_stamp: "Formatted at 2026-09-23 18:11:02 on dist-test-slave-wsx5"
server_key: "8a6cdc830a2f039e566bd002e19a54f4"
server_key_iv: "888d4b4adc442e29d7f0b13717f3aa28"
server_key_version: "encryptionkey@0"
I20260923 18:11:02.746071 31520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:02.758288 31520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:02.761664 31520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:02.761847 31520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:02.768776 31520 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:41757
I20260923 18:11:02.768839 31587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:41757 every 8 connection(s)
I20260923 18:11:02.769809 31520 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:02.772673 31588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:02.775588 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 31520
I20260923 18:11:02.775714 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:02.775945 27951 external_mini_cluster.cc:1428] Setting key a046f6a9200529b47c41fa28cbb07ede
I20260923 18:11:02.778676 31588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9: Bootstrap starting.
I20260923 18:11:02.780905 31588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:02.781674 31588 log.cc:824] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:02.783825 31588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9: No bootstrap required, opened a new log
Sep 23 18:11:02 dist-test-slave-wsx5 krb5kdc[31507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:02.787676 31588 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa232b075d1242238e82df40df8b66d9" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41757 } }
I20260923 18:11:02.787981 31588 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:02.788039 31588 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa232b075d1242238e82df40df8b66d9, State: Initialized, Role: FOLLOWER
I20260923 18:11:02.788553 31588 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa232b075d1242238e82df40df8b66d9" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41757 } }
I20260923 18:11:02.788691 31588 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:02.788750 31588 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:02.788851 31588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:02.789812 31588 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa232b075d1242238e82df40df8b66d9" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41757 } }
I20260923 18:11:02.790169 31588 leader_election.cc:302] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa232b075d1242238e82df40df8b66d9; no voters:
I20260923 18:11:02.790480 31588 leader_election.cc:288] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:02.790660 31593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:02.790872 31593 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [term 1 LEADER]: Becoming Leader. State: Replica: fa232b075d1242238e82df40df8b66d9, State: Running, Role: LEADER
I20260923 18:11:02.791198 31593 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa232b075d1242238e82df40df8b66d9" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41757 } }
I20260923 18:11:02.791684 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:02.777426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.27.75.254:41757)
0923 18:11:02.777894 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:02.777910 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:02.777954 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:11:02.778614 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:02.778632 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:02.779014 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:02.779371 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:02.780149 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:02.781235 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:02.782176 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:02.782850 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:02.783016 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:02.785500 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:02.785508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:02.785529 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:02.785565 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:02.788296 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:02.788855 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:02.788860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:02.788865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:02.789019 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:02.789330 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:02.789333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:02.789335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:02.789794 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:02.789993 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:02.790325 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 18:11:02.790657 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":134,"threads_started":1}
I20260923 18:11:02.791814 31588 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:02.792410 31593 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa232b075d1242238e82df40df8b66d9. Latest consensus state: current_term: 1 leader_uuid: "fa232b075d1242238e82df40df8b66d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa232b075d1242238e82df40df8b66d9" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41757 } } }
I20260923 18:11:02.792510 31593 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:02.793054 31597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:02.794379 31594 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa232b075d1242238e82df40df8b66d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa232b075d1242238e82df40df8b66d9" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41757 } } }
I20260923 18:11:02.794484 31594 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:02.796170 31597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:02.802165 31597 catalog_manager.cc:1347] Generated new cluster ID: 44a9c304f90441a9a61393cbd9f4d984
I20260923 18:11:02.802239 31597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:02.817556 31597 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:02.818946 31597 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:02.833523 31597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa232b075d1242238e82df40df8b66d9: Generated new TSK 0
I20260923 18:11:02.834446 31597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:11:02.864219 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41757
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:02.977900 31614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:02.978180 31614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:02.978231 31614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:02.982463 31614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:02.982534 31614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:02.982595 31614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:02.988237 31614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:02.989459 31614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:02.990509 31614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:03.000669 31620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:03.000660 31619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:03.000726 31614 server_base.cc:1034] running on GCE node
W20260923 18:11:03.000660 31622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:03.001441 31614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:03.002231 31614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:03.003413 31614 hybrid_clock.cc:648] HybridClock initialized: now 1790187063003388 us; error 39 us; skew 500 ppm
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[31507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187063, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:03.006371 31614 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:11:03.007331 31614 webserver.cc:460] Webserver started at http://127.27.75.193:40025/ using document root <none> and password file <none>
I20260923 18:11:03.007859 31614 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:03.007920 31614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:03.008087 31614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:03.010315 31614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "f118a642b7bd40da844aa8871f5f38e9"
format_stamp: "Formatted at 2026-09-23 18:11:03 on dist-test-slave-wsx5"
server_key: "da4abcdb062a901a6e09f52f0ce9aa2f"
server_key_iv: "85eb363a9d681cdc65be9da82a42f173"
server_key_version: "encryptionkey@0"
I20260923 18:11:03.010907 31614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "f118a642b7bd40da844aa8871f5f38e9"
format_stamp: "Formatted at 2026-09-23 18:11:03 on dist-test-slave-wsx5"
server_key: "da4abcdb062a901a6e09f52f0ce9aa2f"
server_key_iv: "85eb363a9d681cdc65be9da82a42f173"
server_key_version: "encryptionkey@0"
I20260923 18:11:03.014763 31614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 18:11:03.017445 31629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:03.018899 31614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:03.019016 31614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "f118a642b7bd40da844aa8871f5f38e9"
format_stamp: "Formatted at 2026-09-23 18:11:03 on dist-test-slave-wsx5"
server_key: "da4abcdb062a901a6e09f52f0ce9aa2f"
server_key_iv: "85eb363a9d681cdc65be9da82a42f173"
server_key_version: "encryptionkey@0"
I20260923 18:11:03.019120 31614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:03.030645 31614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:03.033860 31614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:03.034035 31614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:03.034631 31614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:03.035456 31614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:03.035501 31614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:03.035543 31614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:03.035573 31614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:03.043973 31614 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:39191
I20260923 18:11:03.044075 31742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:39191 every 8 connection(s)
I20260923 18:11:03.044881 31614 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:03.050498 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 31614
I20260923 18:11:03.050627 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:11:03.050977 27951 external_mini_cluster.cc:1428] Setting key f06096f12c00ba304423df0526c38005
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[31507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187063, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:03.059186 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:03.046689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:57273 (local address 127.27.75.254:41757)
0923 18:11:03.046829 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:03.046834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:03.047678 (+ 844us) server_negotiation.cc:407] Connection header received
0923 18:11:03.049055 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:03.049057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:03.049108 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:03.049245 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:03.050491 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.051258 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.052456 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.052680 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.052755 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:03.055464 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:03.055468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:03.055471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:03.055500 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:03.057102 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:03.057613 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:03.057616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:03.057618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:03.057720 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:03.058022 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:03.058025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:03.058027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:03.058295 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:03.058402 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:03.058871 (+ 469us) server_negotiation.cc:299] Negotiation successful
0923 18:11:03.059005 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:11:03.059948 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:03.047103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:41757 (local address 127.27.75.193:57273)
0923 18:11:03.047531 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:03.047571 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:03.048844 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:03.049441 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:03.049449 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:03.049816 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:03.050336 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:03.050350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.051415 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.051418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:03.052322 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:03.052332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.052830 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.052835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:03.053025 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:03.053736 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:03.053756 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:03.055285 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:03.057247 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:03.057252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:03.057265 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:03.057517 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:03.057805 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:03.057807 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:03.057809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:03.057939 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:03.058385 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:03.058390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:03.058630 (+ 240us) client_negotiation.cc:769] Sending connection context
0923 18:11:03.058794 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 18:11:03.059053 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:11:03.061287 31743 heartbeater.cc:347] Connected to a master server at 127.27.75.254:41757
I20260923 18:11:03.061520 31743 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:03.062044 31743 heartbeater.cc:511] Master 127.27.75.254:41757 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:11:03.063664 31552 ts_manager.cc:194] Registered new tserver with Master: f118a642b7bd40da844aa8871f5f38e9 (127.27.75.193:39191)
I20260923 18:11:03.065234 31552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:57273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:11:03.080662 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41757
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:03.190011 31748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:03.190276 31748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:03.190318 31748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:03.194540 31748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:03.194609 31748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:03.194675 31748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:03.200129 31748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:03.201231 31748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:03.202250 31748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:03.212389 31753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:03.212466 31748 server_base.cc:1034] running on GCE node
W20260923 18:11:03.212409 31756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:03.212428 31754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:03.213238 31748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:03.213907 31748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:03.215090 31748 hybrid_clock.cc:648] HybridClock initialized: now 1790187063215044 us; error 58 us; skew 500 ppm
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[31507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187063, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:03.218158 31748 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:11:03.219096 31748 webserver.cc:460] Webserver started at http://127.27.75.194:40161/ using document root <none> and password file <none>
I20260923 18:11:03.219635 31748 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:03.219746 31748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:03.219970 31748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:03.222050 31748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "9e65d7986bd94e52a666bfa5878504ae"
format_stamp: "Formatted at 2026-09-23 18:11:03 on dist-test-slave-wsx5"
server_key: "2bb35e065004069744bc7b5b6ad4fcd2"
server_key_iv: "ad82defbde0b6fcf5f64c4ec1d447be8"
server_key_version: "encryptionkey@0"
I20260923 18:11:03.222595 31748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "9e65d7986bd94e52a666bfa5878504ae"
format_stamp: "Formatted at 2026-09-23 18:11:03 on dist-test-slave-wsx5"
server_key: "2bb35e065004069744bc7b5b6ad4fcd2"
server_key_iv: "ad82defbde0b6fcf5f64c4ec1d447be8"
server_key_version: "encryptionkey@0"
I20260923 18:11:03.226392 31748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260923 18:11:03.228946 31763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:03.230243 31748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 18:11:03.230368 31748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "9e65d7986bd94e52a666bfa5878504ae"
format_stamp: "Formatted at 2026-09-23 18:11:03 on dist-test-slave-wsx5"
server_key: "2bb35e065004069744bc7b5b6ad4fcd2"
server_key_iv: "ad82defbde0b6fcf5f64c4ec1d447be8"
server_key_version: "encryptionkey@0"
I20260923 18:11:03.230477 31748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:03.251516 31748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:03.255024 31748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:03.255226 31748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:03.255818 31748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:03.256656 31748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:03.256702 31748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:03.256755 31748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:03.256781 31748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:03.265741 31748 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:35315
I20260923 18:11:03.265761 31876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:35315 every 8 connection(s)
I20260923 18:11:03.266680 31748 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:03.267021 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 31748
I20260923 18:11:03.267139 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:11:03.267470 27951 external_mini_cluster.cc:1428] Setting key 0199742c7a2e2cbd6e96517140fed6f8
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[31507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187063, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:03.283444 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:03.268711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:50799 (local address 127.27.75.254:41757)
0923 18:11:03.268878 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:03.268883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:03.270029 (+ 1146us) server_negotiation.cc:407] Connection header received
0923 18:11:03.271624 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:03.271627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:03.271688 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:03.271821 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:03.273026 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.273644 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.274820 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.275052 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.275142 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:03.279979 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:03.279982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:03.279984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:03.280014 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:03.281386 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:03.281916 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:03.281918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:03.281920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:03.281998 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:03.282334 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:03.282337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:03.282339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:03.282534 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:03.282623 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:03.283145 (+ 522us) server_negotiation.cc:299] Negotiation successful
0923 18:11:03.283279 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 18:11:03.284287 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:03.269279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:41757 (local address 127.27.75.194:50799)
0923 18:11:03.269870 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:03.269918 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:03.271304 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:03.271972 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:03.271980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:03.272365 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:03.272873 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:03.272887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.273773 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.273775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:03.274689 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:03.274700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.275200 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.275203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:03.275375 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:03.275990 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:03.276006 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:03.279774 (+ 3768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:03.281510 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:03.281517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:03.281536 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:03.281814 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:03.282082 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:03.282085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:03.282087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:03.282236 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:03.282634 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:03.282642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:03.282909 (+ 267us) client_negotiation.cc:769] Sending connection context
0923 18:11:03.283089 (+ 180us) client_negotiation.cc:240] Negotiation successful
0923 18:11:03.283462 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":163,"threads_started":1}
I20260923 18:11:03.285995 31877 heartbeater.cc:347] Connected to a master server at 127.27.75.254:41757
I20260923 18:11:03.286273 31877 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:03.286798 31877 heartbeater.cc:511] Master 127.27.75.254:41757 requested a full tablet report, sending...
I20260923 18:11:03.287961 31552 ts_manager.cc:194] Registered new tserver with Master: 9e65d7986bd94e52a666bfa5878504ae (127.27.75.194:35315)
I20260923 18:11:03.288586 31552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:50799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:11:03.297760 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41757
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:03.411952 31882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:03.412212 31882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:03.412256 31882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:03.416395 31882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:03.416462 31882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:03.416527 31882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:03.421908 31882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:41757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:03.423009 31882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:03.424062 31882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:11:03.433908 31882 server_base.cc:1034] running on GCE node
W20260923 18:11:03.433902 31888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:03.434139 31887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:03.434198 31890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:03.434657 31882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:03.435432 31882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:03.436614 31882 hybrid_clock.cc:648] HybridClock initialized: now 1790187063436578 us; error 48 us; skew 500 ppm
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[31507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187063, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:03.442817 31882 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:11:03.443811 31882 webserver.cc:460] Webserver started at http://127.27.75.195:43745/ using document root <none> and password file <none>
I20260923 18:11:03.444406 31882 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:03.444478 31882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:03.444684 31882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:03.446771 31882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "9279dfc2b33048fc960f788f13d69c98"
format_stamp: "Formatted at 2026-09-23 18:11:03 on dist-test-slave-wsx5"
server_key: "6af41e0cb6413a157f91af2d6f38938f"
server_key_iv: "06db46d3f4480154bd893eb36aae6b97"
server_key_version: "encryptionkey@0"
I20260923 18:11:03.447304 31882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "9279dfc2b33048fc960f788f13d69c98"
format_stamp: "Formatted at 2026-09-23 18:11:03 on dist-test-slave-wsx5"
server_key: "6af41e0cb6413a157f91af2d6f38938f"
server_key_iv: "06db46d3f4480154bd893eb36aae6b97"
server_key_version: "encryptionkey@0"
I20260923 18:11:03.450968 31882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 18:11:03.453411 31897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:03.454624 31882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:11:03.454734 31882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "9279dfc2b33048fc960f788f13d69c98"
format_stamp: "Formatted at 2026-09-23 18:11:03 on dist-test-slave-wsx5"
server_key: "6af41e0cb6413a157f91af2d6f38938f"
server_key_iv: "06db46d3f4480154bd893eb36aae6b97"
server_key_version: "encryptionkey@0"
I20260923 18:11:03.454849 31882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:03.468449 31882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:03.471418 31882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:03.471590 31882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:03.472167 31882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:03.472972 31882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:03.473018 31882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:03.473070 31882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:03.473098 31882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:03.481418 31882 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:35711
I20260923 18:11:03.481451 32010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:35711 every 8 connection(s)
I20260923 18:11:03.482302 31882 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:03.484026 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 31882
I20260923 18:11:03.484149 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:11:03.484485 27951 external_mini_cluster.cc:1428] Setting key 40de34269c6b103f55bb85074512b9a5
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[31507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187063, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:03.495544 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:03.484090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:33217 (local address 127.27.75.254:41757)
0923 18:11:03.484246 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:03.484250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:03.485295 (+ 1045us) server_negotiation.cc:407] Connection header received
0923 18:11:03.486738 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:03.486740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:03.486790 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:03.486857 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:03.487948 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.488466 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.489584 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.489809 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.489884 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:03.492223 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:03.492244 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:03.492247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:03.492282 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:03.493663 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:03.494139 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:03.494143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:03.494145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:03.494221 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:03.494517 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:03.494520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:03.494522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:03.494701 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:03.494781 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:03.495254 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 18:11:03.495382 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:11:03.496323 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:03.484596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:41757 (local address 127.27.75.195:33217)
0923 18:11:03.485102 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:03.485178 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:03.486564 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:03.487004 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:03.487010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:03.487373 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:03.487840 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:03.487852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.488551 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.488554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:03.489483 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:03.489492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.489924 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.489927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:03.490065 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:03.490610 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:03.490625 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:03.492071 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:03.493797 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:03.493803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:03.493817 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:03.494049 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:03.494297 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:03.494300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:03.494302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:03.494432 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:03.494798 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:03.494805 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:03.495040 (+ 235us) client_negotiation.cc:769] Sending connection context
0923 18:11:03.495202 (+ 162us) client_negotiation.cc:240] Negotiation successful
0923 18:11:03.495516 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":126,"threads_started":1}
I20260923 18:11:03.497607 32011 heartbeater.cc:347] Connected to a master server at 127.27.75.254:41757
I20260923 18:11:03.497908 32011 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:03.498431 32011 heartbeater.cc:511] Master 127.27.75.254:41757 requested a full tablet report, sending...
I20260923 18:11:03.499395 31552 ts_manager.cc:194] Registered new tserver with Master: 9279dfc2b33048fc960f788f13d69c98 (127.27.75.195:35711)
I20260923 18:11:03.499948 31552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:33217
I20260923 18:11:03.510347 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[31507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[31507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:11:03.536487 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:03.526479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39800 (local address 127.27.75.193:39191)
0923 18:11:03.526830 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:03.526846 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:03.526872 (+ 26us) server_negotiation.cc:407] Connection header received
0923 18:11:03.526923 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:03.526928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:03.527114 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:03.527235 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:03.528048 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.528931 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.529859 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.530243 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.530333 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:03.532777 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:03.532788 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:03.532801 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:03.532836 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:03.534578 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:03.535084 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:03.535091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:03.535093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:03.535183 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:03.535456 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:03.535460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:03.535462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:03.535739 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:03.535941 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:03.536154 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 18:11:03.536304 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":132,"threads_started":1}
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[31507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:03.547204 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:03.539414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.27.75.254:41757)
0923 18:11:03.539542 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:03.539546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:03.539560 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:03.539672 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:03.539675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:03.539716 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:03.539778 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:03.540366 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.540876 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.541688 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:03.541900 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:03.541986 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:03.544329 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:03.544333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:03.544335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:03.544362 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:03.545728 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:03.546149 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:03.546152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:03.546153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:03.546225 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:03.546451 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:03.546454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:03.546455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:03.546686 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:03.546792 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:03.546989 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 18:11:03.547083 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260923 18:11:03.547607 31552 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55878
I20260923 18:11:03.548197 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 31614
I20260923 18:11:03.553356 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 31748
I20260923 18:11:03.557925 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 31882
I20260923 18:11:03.563351 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 31520
2026-09-23T18:11:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2073 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[32023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[32023](info): set up 2 sockets
Sep 23 18:11:03 dist-test-slave-wsx5 krb5kdc[32023](info): commencing operation
krb5kdc: starting...
W20260923 18:11:04.519912 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:11:04 dist-test-slave-wsx5 krb5kdc[32023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:11:04.640708 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:37379
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:37379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:04.753399 32036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:04.753726 32036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:04.753800 32036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:04.758100 32036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:04.758177 32036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:04.758205 32036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:04.758227 32036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:04.758248 32036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:04.763932 32036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:37379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:37379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:04.765053 32036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:04.766155 32036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:04.774827 32041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:04.774907 32036 server_base.cc:1034] running on GCE node
W20260923 18:11:04.774856 32042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:04.774844 32044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:04.775671 32036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:04.776912 32036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:04.778141 32036 hybrid_clock.cc:648] HybridClock initialized: now 1790187064778117 us; error 37 us; skew 500 ppm
Sep 23 18:11:04 dist-test-slave-wsx5 krb5kdc[32023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187064, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:04.787379 32036 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:11:04.788237 32036 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:04.788317 32036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:04.788530 32036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:04.790786 32036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "42969daa26f34c6c97c243227a927045"
format_stamp: "Formatted at 2026-09-23 18:11:04 on dist-test-slave-wsx5"
server_key: "51113f601ac8660c6c4efcad0302e32e"
server_key_iv: "add1ec4c73bf0aea8521c5cb08ab4e3c"
server_key_version: "encryptionkey@0"
I20260923 18:11:04.791378 32036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "42969daa26f34c6c97c243227a927045"
format_stamp: "Formatted at 2026-09-23 18:11:04 on dist-test-slave-wsx5"
server_key: "51113f601ac8660c6c4efcad0302e32e"
server_key_iv: "add1ec4c73bf0aea8521c5cb08ab4e3c"
server_key_version: "encryptionkey@0"
I20260923 18:11:04.795382 32036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:11:04.798027 32050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:04.799367 32036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:04.799497 32036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "42969daa26f34c6c97c243227a927045"
format_stamp: "Formatted at 2026-09-23 18:11:04 on dist-test-slave-wsx5"
server_key: "51113f601ac8660c6c4efcad0302e32e"
server_key_iv: "add1ec4c73bf0aea8521c5cb08ab4e3c"
server_key_version: "encryptionkey@0"
I20260923 18:11:04.799609 32036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:04.819268 32036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:04.823714 32036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:04.823908 32036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:04.830590 32036 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:37379
I20260923 18:11:04.830830 32102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:37379 every 8 connection(s)
I20260923 18:11:04.831270 32036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:04.834406 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:04.837414 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 32036
I20260923 18:11:04.837527 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:04.837809 27951 external_mini_cluster.cc:1428] Setting key 7b3b154a30e24c264664d6872928c904
I20260923 18:11:04.840796 32103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045: Bootstrap starting.
I20260923 18:11:04.843611 32103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:04.844411 32103 log.cc:824] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:04.846572 32103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045: No bootstrap required, opened a new log
Sep 23 18:11:04 dist-test-slave-wsx5 krb5kdc[32023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:04.849923 32103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42969daa26f34c6c97c243227a927045" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37379 } }
I20260923 18:11:04.850170 32103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:04.850208 32103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42969daa26f34c6c97c243227a927045, State: Initialized, Role: FOLLOWER
I20260923 18:11:04.850682 32103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42969daa26f34c6c97c243227a927045" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37379 } }
I20260923 18:11:04.850852 32103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:04.850921 32103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:04.851024 32103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:04.852072 32103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42969daa26f34c6c97c243227a927045" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37379 } }
I20260923 18:11:04.852454 32103 leader_election.cc:302] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42969daa26f34c6c97c243227a927045; no voters:
I20260923 18:11:04.852846 32103 leader_election.cc:288] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:04.852939 32108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:04.853099 32108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [term 1 LEADER]: Becoming Leader. State: Replica: 42969daa26f34c6c97c243227a927045, State: Running, Role: LEADER
I20260923 18:11:04.853610 32108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42969daa26f34c6c97c243227a927045" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37379 } }
I20260923 18:11:04.853745 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:04.839148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.27.75.254:37379)
0923 18:11:04.839599 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:04.839620 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:04.839666 (+ 46us) server_negotiation.cc:407] Connection header received
0923 18:11:04.840541 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:04.840566 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:04.841015 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:04.841431 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:04.842102 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:04.843604 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:04.844536 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:04.845151 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:04.845289 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:04.847939 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:04.847944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:04.847960 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:04.847989 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:04.850542 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:04.851071 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:04.851076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:04.851082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:04.851230 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:04.851569 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:04.851571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:04.851572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:04.851996 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:04.852138 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:04.852470 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 18:11:04.852781 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":122,"threads_started":1}
I20260923 18:11:04.854377 32103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:04.856632 32108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42969daa26f34c6c97c243227a927045. Latest consensus state: current_term: 1 leader_uuid: "42969daa26f34c6c97c243227a927045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42969daa26f34c6c97c243227a927045" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37379 } } }
I20260923 18:11:04.856769 32109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42969daa26f34c6c97c243227a927045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42969daa26f34c6c97c243227a927045" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37379 } } }
I20260923 18:11:04.856840 32108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:04.856858 32109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045 [sys.catalog]: This master's current role is: LEADER
W20260923 18:11:04.859477 32123 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:11:04.859627 32123 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:11:04.859722 32124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:04.860419 32124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:04.866693 32124 catalog_manager.cc:1347] Generated new cluster ID: 52f44a971d63432cae0eca756edde1cb
I20260923 18:11:04.866786 32124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:04.880714 32124 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:04.881826 32124 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:04.895923 32124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045: Generated new TSK 0
I20260923 18:11:04.896656 32124 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:11:04.927511 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:37379
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:05.040383 32130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:05.040668 32130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:05.040714 32130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:05.044998 32130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:05.045075 32130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:05.045173 32130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:05.050700 32130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:37379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:05.051786 32130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:05.052845 32130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:05.062567 32135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:05.062551 32136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:05.062677 32130 server_base.cc:1034] running on GCE node
W20260923 18:11:05.062552 32138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:05.063277 32130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:05.064036 32130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:05.065203 32130 hybrid_clock.cc:648] HybridClock initialized: now 1790187065065183 us; error 30 us; skew 500 ppm
Sep 23 18:11:05 dist-test-slave-wsx5 krb5kdc[32023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187065, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:05.068135 32130 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:11:05.068915 32130 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:05.068987 32130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:05.069218 32130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:05.071411 32130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "7e5504175a724439a1ec7d6ac10fbe59"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "ee200d74f874d4740bcb9615de156394"
server_key_iv: "f0efe6d99f972c1bf0830134d118a18d"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.071985 32130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "7e5504175a724439a1ec7d6ac10fbe59"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "ee200d74f874d4740bcb9615de156394"
server_key_iv: "f0efe6d99f972c1bf0830134d118a18d"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.076007 32130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:11:05.078627 32144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:05.079917 32130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:11:05.080044 32130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "7e5504175a724439a1ec7d6ac10fbe59"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "ee200d74f874d4740bcb9615de156394"
server_key_iv: "f0efe6d99f972c1bf0830134d118a18d"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.080152 32130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:05.103722 32130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:05.106690 32130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:05.106871 32130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:05.107446 32130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:05.108300 32130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:05.108351 32130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:05.108404 32130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:05.108433 32130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:05.116917 32130 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36637
I20260923 18:11:05.116940 32257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36637 every 8 connection(s)
I20260923 18:11:05.117533 32130 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:05.124135 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 32130
I20260923 18:11:05.124275 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:11:05.124529 27951 external_mini_cluster.cc:1428] Setting key c40a275ed25efe5e21e1bc3ff43f49be
Sep 23 18:11:05 dist-test-slave-wsx5 krb5kdc[32023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187065, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:05.131466 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.119333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:50631 (local address 127.27.75.254:37379)
0923 18:11:05.119511 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:05.119516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:05.120420 (+ 904us) server_negotiation.cc:407] Connection header received
0923 18:11:05.121738 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:05.121741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:05.121794 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:05.121872 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:05.123183 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.123755 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.124970 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.125285 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.125390 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.128018 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:05.128021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:05.128023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:05.128048 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:05.129473 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:05.129961 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:05.129964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:05.129966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:05.130045 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:05.130355 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:05.130358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:05.130360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:05.130549 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:05.130633 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:05.131132 (+ 499us) server_negotiation.cc:299] Negotiation successful
0923 18:11:05.131285 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 18:11:05.132284 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.119756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:37379 (local address 127.27.75.193:50631)
0923 18:11:05.120252 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:05.120296 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:05.121514 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:05.122101 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:05.122108 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:05.122470 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:05.123020 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.123034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.123885 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.123888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.124848 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.124858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.125453 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.125456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.125674 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.126284 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:05.126301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:05.127855 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:05.129584 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:05.129591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:05.129609 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:05.129868 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:05.130124 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:05.130127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:05.130129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:05.130271 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:05.130645 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:05.130652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:05.130906 (+ 254us) client_negotiation.cc:769] Sending connection context
0923 18:11:05.131075 (+ 169us) client_negotiation.cc:240] Negotiation successful
0923 18:11:05.131442 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:11:05.133567 32258 heartbeater.cc:347] Connected to a master server at 127.27.75.254:37379
I20260923 18:11:05.133831 32258 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:05.134298 32258 heartbeater.cc:511] Master 127.27.75.254:37379 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:11:05.135900 32067 ts_manager.cc:194] Registered new tserver with Master: 7e5504175a724439a1ec7d6ac10fbe59 (127.27.75.193:36637)
I20260923 18:11:05.137415 32067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:50631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:11:05.151714 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:37379
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:05.261149 32263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:05.261462 32263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:05.261547 32263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:05.265668 32263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:05.265738 32263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:05.265821 32263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:05.271162 32263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:37379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:05.272236 32263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:05.273272 32263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:05.282961 32271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:05.283041 32268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:05.283262 32269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:05.283264 32263 server_base.cc:1034] running on GCE node
I20260923 18:11:05.283768 32263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:05.284493 32263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:05.285672 32263 hybrid_clock.cc:648] HybridClock initialized: now 1790187065285641 us; error 42 us; skew 500 ppm
Sep 23 18:11:05 dist-test-slave-wsx5 krb5kdc[32023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187065, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:05.288528 32263 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:11:05.289322 32263 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:05.289395 32263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:05.289574 32263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:05.291714 32263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "e15f661b0e1d46b79d5ad93896dfaa78"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "0502bd0edfe82aa4e21ade09782ed3e7"
server_key_iv: "0b4d31d5605d99b96c350c4bc147746c"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.292280 32263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "e15f661b0e1d46b79d5ad93896dfaa78"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "0502bd0edfe82aa4e21ade09782ed3e7"
server_key_iv: "0b4d31d5605d99b96c350c4bc147746c"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.296811 32263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:11:05.301251 32277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:05.302943 32263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:11:05.303090 32263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "e15f661b0e1d46b79d5ad93896dfaa78"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "0502bd0edfe82aa4e21ade09782ed3e7"
server_key_iv: "0b4d31d5605d99b96c350c4bc147746c"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.303236 32263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:05.323908 32263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:05.327811 32263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:05.328023 32263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:05.328645 32263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:05.329561 32263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:05.329609 32263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:05.329663 32263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:05.329691 32263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:05.338295 32263 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:40013
I20260923 18:11:05.338315 32390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:40013 every 8 connection(s)
I20260923 18:11:05.338907 32263 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
Sep 23 18:11:05 dist-test-slave-wsx5 krb5kdc[32023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187065, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:05.348522 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 32263
I20260923 18:11:05.348642 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:11:05.348918 27951 external_mini_cluster.cc:1428] Setting key 2f289724f5c2008ec830f4235204f9cd
I20260923 18:11:05.353473 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.340721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:59327 (local address 127.27.75.254:37379)
0923 18:11:05.340869 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:05.340874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:05.341732 (+ 858us) server_negotiation.cc:407] Connection header received
0923 18:11:05.343032 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:05.343035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:05.343095 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:05.343170 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:05.344334 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.344861 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.346055 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.346290 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.346384 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.349077 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:05.349081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:05.349083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:05.349112 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:05.350689 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:05.351350 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:05.351353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:05.351354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:05.351447 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:05.351896 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:05.351899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:05.351900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:05.352180 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:05.352383 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:05.353162 (+ 779us) server_negotiation.cc:299] Negotiation successful
0923 18:11:05.353310 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 18:11:05.354516 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.341040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:37379 (local address 127.27.75.194:59327)
0923 18:11:05.341565 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:05.341609 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:05.342818 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:05.343348 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:05.343355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:05.343703 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:05.344183 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.344196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.344980 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.344983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.345928 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.345938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.346439 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.346444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.346608 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.347186 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:05.347203 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:05.348856 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:05.350816 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:05.350824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:05.350845 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:05.351219 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:05.351537 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:05.351540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:05.351542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:05.351761 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:05.352396 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:05.352445 (+ 49us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:05.352811 (+ 366us) client_negotiation.cc:769] Sending connection context
0923 18:11:05.353070 (+ 259us) client_negotiation.cc:240] Negotiation successful
0923 18:11:05.353528 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":114,"threads_started":1}
I20260923 18:11:05.355906 32391 heartbeater.cc:347] Connected to a master server at 127.27.75.254:37379
I20260923 18:11:05.356217 32391 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:05.356722 32391 heartbeater.cc:511] Master 127.27.75.254:37379 requested a full tablet report, sending...
I20260923 18:11:05.357961 32067 ts_manager.cc:194] Registered new tserver with Master: e15f661b0e1d46b79d5ad93896dfaa78 (127.27.75.194:40013)
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:11:05.358566 32067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:59327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:11:05.378564 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:37379
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:34703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:05.492038 32396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:05.492316 32396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:05.492360 32396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:05.496572 32396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:05.496645 32396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:05.496711 32396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:05.502336 32396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:34703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:37379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:05.503476 32396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:05.504554 32396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:05.514537 32401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:05.514547 32404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:05.514549 32402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:05.514904 32396 server_base.cc:1034] running on GCE node
I20260923 18:11:05.515477 32396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:05.516157 32396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:05.517329 32396 hybrid_clock.cc:648] HybridClock initialized: now 1790187065517311 us; error 32 us; skew 500 ppm
Sep 23 18:11:05 dist-test-slave-wsx5 krb5kdc[32023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187065, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:05.520258 32396 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:11:05.521190 32396 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:05.521273 32396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:05.521490 32396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:05.523607 32396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "adb5858994af406190bc4701e8f6c176"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "2453f00bcccdbf2458ce7b541d2b26d3"
server_key_iv: "7ea45a9ca61a73e4f74d667693ac1f69"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.524197 32396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "adb5858994af406190bc4701e8f6c176"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "2453f00bcccdbf2458ce7b541d2b26d3"
server_key_iv: "7ea45a9ca61a73e4f74d667693ac1f69"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.528210 32396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 18:11:05.531085 32410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:05.532430 32396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:11:05.532622 32396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "adb5858994af406190bc4701e8f6c176"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "2453f00bcccdbf2458ce7b541d2b26d3"
server_key_iv: "7ea45a9ca61a73e4f74d667693ac1f69"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.532766 32396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:05.552043 32396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:05.555796 32396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:05.555994 32396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:05.556632 32396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:05.557557 32396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:05.557611 32396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:05.557685 32396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:05.557715 32396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:05.566457 32396 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:41739
I20260923 18:11:05.566529 32523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:41739 every 8 connection(s)
I20260923 18:11:05.567112 32396 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:05.575296 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 32396
I20260923 18:11:05.575439 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:11:05.575728 27951 external_mini_cluster.cc:1428] Setting key 0e79da21e6e7950e72e4517e37010cf9
Sep 23 18:11:05 dist-test-slave-wsx5 krb5kdc[32023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187065, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:05.582561 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.569007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:41343 (local address 127.27.75.254:37379)
0923 18:11:05.569282 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:05.569287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:05.570307 (+ 1020us) server_negotiation.cc:407] Connection header received
0923 18:11:05.571711 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:05.571715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:05.571777 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:05.571854 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:05.573144 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.573743 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.574951 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.575282 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.575399 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.578525 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:05.578528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:05.578530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:05.578559 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:05.579976 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:05.580650 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:05.580654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:05.580656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:05.580755 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:05.581262 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:05.581265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:05.581267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:05.581487 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:05.581577 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:05.582298 (+ 721us) server_negotiation.cc:299] Negotiation successful
0923 18:11:05.582428 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 18:11:05.583469 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.569548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:37379 (local address 127.27.75.195:41343)
0923 18:11:05.570130 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:05.570181 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:05.571463 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:05.572036 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:05.572045 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:05.572426 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:05.572957 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.572971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.573881 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.573884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.574819 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.574830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.575533 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.575538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.575711 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.576354 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:05.576372 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:05.578335 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:05.580119 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:05.580125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:05.580141 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:05.580484 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:05.580896 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:05.580899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:05.580900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:05.581152 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:05.581663 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:05.581670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:05.581970 (+ 300us) client_negotiation.cc:769] Sending connection context
0923 18:11:05.582146 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 18:11:05.582455 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":128,"threads_started":1}
I20260923 18:11:05.584800 32524 heartbeater.cc:347] Connected to a master server at 127.27.75.254:37379
I20260923 18:11:05.585049 32524 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:05.585578 32524 heartbeater.cc:511] Master 127.27.75.254:37379 requested a full tablet report, sending...
I20260923 18:11:05.586704 32067 ts_manager.cc:194] Registered new tserver with Master: adb5858994af406190bc4701e8f6c176 (127.27.75.195:41739)
I20260923 18:11:05.587580 32067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:41343
I20260923 18:11:05.590164 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:11:05.600236 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.592183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.27.75.254:37379)
0923 18:11:05.592317 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:05.592322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:05.592500 (+ 178us) server_negotiation.cc:407] Connection header received
0923 18:11:05.592621 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:05.592624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:05.592672 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:05.592736 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:05.593527 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.594067 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.594904 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.595127 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.595215 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.596674 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:05.596679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:05.596682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:05.596713 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:05.598390 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:05.598930 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:05.598933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:05.598934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:05.599031 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:05.599310 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:05.599313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:05.599315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:05.599557 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:05.599689 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:05.599931 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 18:11:05.600069 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:11:05.603427 32067 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:11:05.606604 32067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:11:05.621706 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.615364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:40013 (local address 127.0.0.1:57968)
0923 18:11:05.616046 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:05.616059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:05.616139 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:05.617339 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:05.617342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:05.617360 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:05.617472 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.617478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.618780 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.618785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.620028 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.620038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.621169 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.621172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.621284 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.621307 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:11:05.621377 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 18:11:05.621528 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":55,"thread_start_us":86,"threads_started":1}
I20260923 18:11:05.621776 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.616071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.27.75.194:40013)
0923 18:11:05.616843 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:05.616854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:05.616878 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:11:05.616950 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:05.616956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:05.617195 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:05.617353 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:05.617601 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.618557 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.620312 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.621014 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.621185 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.621331 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:05.621503 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 18:11:05.621607 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":75,"threads_started":1}
I20260923 18:11:05.621764 32537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.615451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.27.75.193:36637)
0923 18:11:05.615869 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:05.615876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:05.615901 (+ 25us) server_negotiation.cc:407] Connection header received
0923 18:11:05.616118 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:05.616124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:05.616349 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:05.616512 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:05.616996 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.618125 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.619820 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.620726 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.620843 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.621010 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:05.621268 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 18:11:05.621513 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":115,"threads_started":1}
I20260923 18:11:05.621704 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.616022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:41739 (local address 127.0.0.1:60290)
0923 18:11:05.616466 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:05.616478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:05.616563 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:05.617532 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:05.617535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:05.617552 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:05.617641 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.617647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.618968 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.618971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.620084 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.620096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.621350 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.621353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.621439 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.621458 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 18:11:05.621517 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 18:11:05.621570 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":93,"threads_started":1}
I20260923 18:11:05.622198 32535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.615281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36637 (local address 127.0.0.1:51954)
0923 18:11:05.615707 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:05.615762 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:05.615898 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:05.616683 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:05.616702 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:05.616724 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:05.616864 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.616871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.618278 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.618282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.619666 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.619678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.620865 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.620869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.621007 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.621060 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 18:11:05.621966 (+ 906us) client_negotiation.cc:240] Negotiation successful
0923 18:11:05.622059 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260923 18:11:05.622690 32540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.616431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.27.75.195:41739)
0923 18:11:05.616892 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:05.616905 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:05.616927 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:11:05.617001 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:05.617006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:05.617241 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:05.617388 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:05.617763 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.618818 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.620227 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.621026 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.621149 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.621291 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:05.621775 (+ 484us) server_negotiation.cc:299] Negotiation successful
0923 18:11:05.622562 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":140,"threads_started":1}
I20260923 18:11:05.625032 32325 tablet_service.cc:1467] Processing CreateTablet for tablet 54c1e091edae46e09dbbd777df660eb0 (DEFAULT_TABLE table=test-table [id=f89ac7e3453e45848e618c610a929d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:05.625293 32458 tablet_service.cc:1467] Processing CreateTablet for tablet 54c1e091edae46e09dbbd777df660eb0 (DEFAULT_TABLE table=test-table [id=f89ac7e3453e45848e618c610a929d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:05.625314 32192 tablet_service.cc:1467] Processing CreateTablet for tablet 54c1e091edae46e09dbbd777df660eb0 (DEFAULT_TABLE table=test-table [id=f89ac7e3453e45848e618c610a929d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:05.626358 32325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c1e091edae46e09dbbd777df660eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:05.626389 32192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c1e091edae46e09dbbd777df660eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:05.626407 32458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c1e091edae46e09dbbd777df660eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:05.633397 32541 tablet_bootstrap.cc:492] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78: Bootstrap starting.
I20260923 18:11:05.634703 32543 tablet_bootstrap.cc:492] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176: Bootstrap starting.
I20260923 18:11:05.635697 32541 tablet_bootstrap.cc:654] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:05.636191 32542 tablet_bootstrap.cc:492] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59: Bootstrap starting.
I20260923 18:11:05.636641 32543 tablet_bootstrap.cc:654] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:05.636854 32541 log.cc:824] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:05.637984 32543 log.cc:824] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:05.638438 32542 tablet_bootstrap.cc:654] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:05.639117 32541 tablet_bootstrap.cc:492] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78: No bootstrap required, opened a new log
I20260923 18:11:05.639253 32542 log.cc:824] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:05.639324 32541 ts_tablet_manager.cc:1397] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:11:05.639964 32543 tablet_bootstrap.cc:492] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176: No bootstrap required, opened a new log
I20260923 18:11:05.640152 32543 ts_tablet_manager.cc:1397] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 18:11:05.641330 32542 tablet_bootstrap.cc:492] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59: No bootstrap required, opened a new log
I20260923 18:11:05.641517 32542 ts_tablet_manager.cc:1397] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260923 18:11:05.642550 32541 raft_consensus.cc:348] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5504175a724439a1ec7d6ac10fbe59" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36637 } } peers { permanent_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40013 } } peers { permanent_uuid: "adb5858994af406190bc4701e8f6c176" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41739 } }
I20260923 18:11:05.642791 32541 raft_consensus.cc:374] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:05.642843 32541 raft_consensus.cc:729] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15f661b0e1d46b79d5ad93896dfaa78, State: Initialized, Role: FOLLOWER
I20260923 18:11:05.643278 32541 consensus_queue.cc:260] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5504175a724439a1ec7d6ac10fbe59" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36637 } } peers { permanent_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40013 } } peers { permanent_uuid: "adb5858994af406190bc4701e8f6c176" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41739 } }
I20260923 18:11:05.643922 32391 heartbeater.cc:503] Master 127.27.75.254:37379 was elected leader, sending a full tablet report...
I20260923 18:11:05.643965 32543 raft_consensus.cc:348] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5504175a724439a1ec7d6ac10fbe59" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36637 } } peers { permanent_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40013 } } peers { permanent_uuid: "adb5858994af406190bc4701e8f6c176" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41739 } }
I20260923 18:11:05.644212 32543 raft_consensus.cc:374] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:05.644263 32543 raft_consensus.cc:729] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb5858994af406190bc4701e8f6c176, State: Initialized, Role: FOLLOWER
I20260923 18:11:05.644277 32541 ts_tablet_manager.cc:1428] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 18:11:05.644731 32543 consensus_queue.cc:260] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5504175a724439a1ec7d6ac10fbe59" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36637 } } peers { permanent_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40013 } } peers { permanent_uuid: "adb5858994af406190bc4701e8f6c176" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41739 } }
I20260923 18:11:05.645438 32524 heartbeater.cc:503] Master 127.27.75.254:37379 was elected leader, sending a full tablet report...
I20260923 18:11:05.646039 32542 raft_consensus.cc:348] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5504175a724439a1ec7d6ac10fbe59" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36637 } } peers { permanent_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40013 } } peers { permanent_uuid: "adb5858994af406190bc4701e8f6c176" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41739 } }
I20260923 18:11:05.646307 32542 raft_consensus.cc:374] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:05.646364 32542 raft_consensus.cc:729] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5504175a724439a1ec7d6ac10fbe59, State: Initialized, Role: FOLLOWER
I20260923 18:11:05.646888 32543 ts_tablet_manager.cc:1428] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 18:11:05.646852 32542 consensus_queue.cc:260] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5504175a724439a1ec7d6ac10fbe59" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36637 } } peers { permanent_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40013 } } peers { permanent_uuid: "adb5858994af406190bc4701e8f6c176" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41739 } }
I20260923 18:11:05.647475 32258 heartbeater.cc:503] Master 127.27.75.254:37379 was elected leader, sending a full tablet report...
I20260923 18:11:05.647769 32542 ts_tablet_manager.cc:1428] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:11:05.669005 32548 raft_consensus.cc:482] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:11:05.669245 32548 raft_consensus.cc:504] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5504175a724439a1ec7d6ac10fbe59" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36637 } } peers { permanent_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40013 } } peers { permanent_uuid: "adb5858994af406190bc4701e8f6c176" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41739 } }
I20260923 18:11:05.670430 32548 leader_election.cc:288] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5504175a724439a1ec7d6ac10fbe59 (127.27.75.193:36637), e15f661b0e1d46b79d5ad93896dfaa78 (127.27.75.194:40013)
I20260923 18:11:05.675438 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.670650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36637 (local address 127.27.75.195:53097)
0923 18:11:05.670840 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:05.670854 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:05.670939 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:05.671232 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:05.671235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:05.671251 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:05.671361 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.671368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.672473 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.672476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.673920 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.673932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.674888 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.674892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.675057 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.675091 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 18:11:05.675176 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 18:11:05.675259 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":101}
I20260923 18:11:05.675463 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.670831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:57693 (local address 127.27.75.194:40013)
0923 18:11:05.670953 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:05.670957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:05.671177 (+ 220us) server_negotiation.cc:407] Connection header received
0923 18:11:05.671262 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:05.671264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:05.671311 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:05.671368 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:05.671601 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.672467 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.674085 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.674736 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.674876 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.674932 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:05.675242 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 18:11:05.675330 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:11:05.675463 32537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.670768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:53097 (local address 127.27.75.193:36637)
0923 18:11:05.670872 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:05.670876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:05.670906 (+ 30us) server_negotiation.cc:407] Connection header received
0923 18:11:05.671067 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:05.671070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:05.671116 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:05.671195 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:05.671471 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.672339 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.674085 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.674763 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.674876 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.674932 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:05.675242 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 18:11:05.675336 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 18:11:05.675447 32550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.670714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:40013 (local address 127.27.75.195:57693)
0923 18:11:05.671094 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:05.671104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:05.671180 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:05.671388 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:05.671391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:05.671405 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:05.671498 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.671508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.672581 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.672584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.673918 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:05.673932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.674888 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.674892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:05.675045 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.675091 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 18:11:05.675176 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 18:11:05.675260 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":17,"thread_start_us":115,"threads_started":1}
I20260923 18:11:05.676138 32212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c1e091edae46e09dbbd777df660eb0" candidate_uuid: "adb5858994af406190bc4701e8f6c176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5504175a724439a1ec7d6ac10fbe59" is_pre_election: true
I20260923 18:11:05.676127 32345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c1e091edae46e09dbbd777df660eb0" candidate_uuid: "adb5858994af406190bc4701e8f6c176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" is_pre_election: true
I20260923 18:11:05.676430 32345 raft_consensus.cc:2454] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adb5858994af406190bc4701e8f6c176 in term 0.
I20260923 18:11:05.676429 32212 raft_consensus.cc:2454] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adb5858994af406190bc4701e8f6c176 in term 0.
I20260923 18:11:05.676956 32413 leader_election.cc:302] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adb5858994af406190bc4701e8f6c176, e15f661b0e1d46b79d5ad93896dfaa78; no voters:
I20260923 18:11:05.677306 32548 raft_consensus.cc:2781] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:11:05.677496 32548 raft_consensus.cc:482] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:11:05.677569 32548 raft_consensus.cc:3037] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:05.678680 32548 raft_consensus.cc:504] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5504175a724439a1ec7d6ac10fbe59" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36637 } } peers { permanent_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40013 } } peers { permanent_uuid: "adb5858994af406190bc4701e8f6c176" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41739 } }
I20260923 18:11:05.679019 32548 leader_election.cc:288] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [CANDIDATE]: Term 1 election: Requested vote from peers 7e5504175a724439a1ec7d6ac10fbe59 (127.27.75.193:36637), e15f661b0e1d46b79d5ad93896dfaa78 (127.27.75.194:40013)
I20260923 18:11:05.679414 32212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c1e091edae46e09dbbd777df660eb0" candidate_uuid: "adb5858994af406190bc4701e8f6c176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5504175a724439a1ec7d6ac10fbe59"
I20260923 18:11:05.679500 32345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c1e091edae46e09dbbd777df660eb0" candidate_uuid: "adb5858994af406190bc4701e8f6c176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e15f661b0e1d46b79d5ad93896dfaa78"
I20260923 18:11:05.679553 32212 raft_consensus.cc:3037] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:05.679618 32345 raft_consensus.cc:3037] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:05.680748 32212 raft_consensus.cc:2454] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adb5858994af406190bc4701e8f6c176 in term 1.
I20260923 18:11:05.680886 32345 raft_consensus.cc:2454] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adb5858994af406190bc4701e8f6c176 in term 1.
I20260923 18:11:05.681229 32413 leader_election.cc:302] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adb5858994af406190bc4701e8f6c176, e15f661b0e1d46b79d5ad93896dfaa78; no voters:
I20260923 18:11:05.681416 32548 raft_consensus.cc:2781] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:05.681627 32548 raft_consensus.cc:686] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 1 LEADER]: Becoming Leader. State: Replica: adb5858994af406190bc4701e8f6c176, State: Running, Role: LEADER
I20260923 18:11:05.681950 32548 consensus_queue.cc:237] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5504175a724439a1ec7d6ac10fbe59" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36637 } } peers { permanent_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40013 } } peers { permanent_uuid: "adb5858994af406190bc4701e8f6c176" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41739 } }
I20260923 18:11:05.685421 32065 catalog_manager.cc:5495] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 reported cstate change: term changed from 0 to 1, leader changed from <none> to adb5858994af406190bc4701e8f6c176 (127.27.75.195). New cstate: current_term: 1 leader_uuid: "adb5858994af406190bc4701e8f6c176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5504175a724439a1ec7d6ac10fbe59" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adb5858994af406190bc4701e8f6c176" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41739 } health_report { overall_health: HEALTHY } } }
I20260923 18:11:05.704919 32540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.700995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.27.75.195:41739)
0923 18:11:05.701145 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:05.701150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:05.701165 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:11:05.701327 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:05.701329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:05.701387 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:05.701482 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:11:05.701743 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.702257 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.703155 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:05.703409 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:05.703499 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:05.703987 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:11:05.704463 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:11:05.704590 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:05.704661 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 18:11:05.704741 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 18:11:05.711453 32345 raft_consensus.cc:1261] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78 [term 1 FOLLOWER]: Refusing update from remote peer adb5858994af406190bc4701e8f6c176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:05.711453 32212 raft_consensus.cc:1261] T 54c1e091edae46e09dbbd777df660eb0 P 7e5504175a724439a1ec7d6ac10fbe59 [term 1 FOLLOWER]: Refusing update from remote peer adb5858994af406190bc4701e8f6c176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:05.712212 32548 consensus_queue.cc:1035] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5504175a724439a1ec7d6ac10fbe59" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:05.712423 32551 consensus_queue.cc:1035] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e15f661b0e1d46b79d5ad93896dfaa78" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:05.720572 32556 mvcc.cc:205] Tried to move back new op lower bound from 7332606221146603520 to 7332606221036179456. Current Snapshot: MvccSnapshot[applied={T|T < 7332606221146603520}]
I20260923 18:11:05.730011 32065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:05.730248 32065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:05.733731 32065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42969daa26f34c6c97c243227a927045: Sending DeleteTablet for 3 replicas of tablet 54c1e091edae46e09dbbd777df660eb0
I20260923 18:11:05.734813 32325 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c1e091edae46e09dbbd777df660eb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:05 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:57968
I20260923 18:11:05.734894 32192 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c1e091edae46e09dbbd777df660eb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:05 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:51954
I20260923 18:11:05.735015 32458 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c1e091edae46e09dbbd777df660eb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:05 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:60290
I20260923 18:11:05.735483 32564 tablet_replica.cc:330] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78: stopping tablet replica
I20260923 18:11:05.735834 32566 tablet_replica.cc:330] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176: stopping tablet replica
I20260923 18:11:05.735867 32564 raft_consensus.cc:2229] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:05.736188 32566 raft_consensus.cc:2229] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:11:05.736212 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 32130
I20260923 18:11:05.736246 32564 raft_consensus.cc:2258] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:05.737426 32566 raft_consensus.cc:2258] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:05.738715 32566 ts_tablet_manager.cc:1905] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:05.738715 32564 ts_tablet_manager.cc:1905] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:05.741365 32564 ts_tablet_manager.cc:1918] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:05.741468 32564 log.cc:1196] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/54c1e091edae46e09dbbd777df660eb0
I20260923 18:11:05.741592 32566 ts_tablet_manager.cc:1918] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:05.741694 32566 log.cc:1196] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/54c1e091edae46e09dbbd777df660eb0
I20260923 18:11:05.741802 32564 ts_tablet_manager.cc:1939] T 54c1e091edae46e09dbbd777df660eb0 P e15f661b0e1d46b79d5ad93896dfaa78: Deleting consensus metadata
I20260923 18:11:05.742014 32566 ts_tablet_manager.cc:1939] T 54c1e091edae46e09dbbd777df660eb0 P adb5858994af406190bc4701e8f6c176: Deleting consensus metadata
I20260923 18:11:05.743328 32051 catalog_manager.cc:4841] TS adb5858994af406190bc4701e8f6c176 (127.27.75.195:41739): tablet 54c1e091edae46e09dbbd777df660eb0 (table test-table [id=f89ac7e3453e45848e618c610a929d93]) successfully deleted
I20260923 18:11:05.743330 32053 catalog_manager.cc:4841] TS e15f661b0e1d46b79d5ad93896dfaa78 (127.27.75.194:40013): tablet 54c1e091edae46e09dbbd777df660eb0 (table test-table [id=f89ac7e3453e45848e618c610a929d93]) successfully deleted
W20260923 18:11:05.747123 32051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:36637 (error 108)
I20260923 18:11:05.747376 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 32263
I20260923 18:11:05.750778 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.750166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36637 (local address 127.0.0.1:51964)
0923 18:11:05.750515 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:05.750615 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 18:11:05.751017 32051 catalog_manager.cc:4584] TS 7e5504175a724439a1ec7d6ac10fbe59 (127.27.75.193:36637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54c1e091edae46e09dbbd777df660eb0: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36637: connect: Connection refused (error 111)
I20260923 18:11:05.755435 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 32396
I20260923 18:11:05.762174 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 32036
2026-09-23T18:11:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2197 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T18:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:05Z Disabled control of system clock
I20260923 18:11:05.793740 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42361
--webserver_interface=127.27.75.254
--webserver_port=0
--builtin_ntp_servers=127.27.75.212:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:42361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 18:11:05.906085 32571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:05.906371 32571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:05.906409 32571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:05.910468 32571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:05.910529 32571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:05.910543 32571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:05.910558 32571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:05.910573 32571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:05.915478 32571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:42361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:05.916541 32571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:05.917904 32571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:05.926415 32579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:05.926522 32576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:05.926646 32571 server_base.cc:1034] running on GCE node
W20260923 18:11:05.926415 32577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:05.927405 32571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:05.928577 32571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:05.929782 32571 hybrid_clock.cc:648] HybridClock initialized: now 1790187065929762 us; error 31 us; skew 500 ppm
I20260923 18:11:05.931805 32571 webserver.cc:460] Webserver started at http://127.27.75.254:38817/ using document root <none> and password file <none>
I20260923 18:11:05.932391 32571 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:05.932454 32571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:05.932627 32571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:05.934921 32571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "10d3e998dd3b4733998dc369897ea9c2"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "54f9deac0a8ab85a0b709adf462260e3"
server_key_iv: "1d714407a8f1ada18cf50c59bb7935fe"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.935534 32571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "10d3e998dd3b4733998dc369897ea9c2"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "54f9deac0a8ab85a0b709adf462260e3"
server_key_iv: "1d714407a8f1ada18cf50c59bb7935fe"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.939633 32571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 18:11:05.942382 32585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:05.943837 32571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:05.943969 32571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "10d3e998dd3b4733998dc369897ea9c2"
format_stamp: "Formatted at 2026-09-23 18:11:05 on dist-test-slave-wsx5"
server_key: "54f9deac0a8ab85a0b709adf462260e3"
server_key_iv: "1d714407a8f1ada18cf50c59bb7935fe"
server_key_version: "encryptionkey@0"
I20260923 18:11:05.944087 32571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:05.948853 32571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:05.949572 32571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:05.949738 32571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:05.956482 32571 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:42361
I20260923 18:11:05.956497 32637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:42361 every 8 connection(s)
I20260923 18:11:05.957455 32571 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:05.959753 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 32571
I20260923 18:11:05.959846 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:05.960100 27951 external_mini_cluster.cc:1428] Setting key 7ed3f48620a09270215ab0f56c084ac9
I20260923 18:11:05.960638 32638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:05.966105 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:05.961443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.27.75.254:42361)
0923 18:11:05.962109 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:05.962129 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:05.962170 (+ 41us) server_negotiation.cc:407] Connection header received
0923 18:11:05.962942 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:05.962967 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:05.963429 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:05.963820 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:05.964527 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:05.964533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:05.964550 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:05.964561 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:05.964626 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:05.964648 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:05.964790 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:05.964908 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 18:11:05.965211 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260923 18:11:05.967723 32638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2: Bootstrap starting.
I20260923 18:11:05.969934 32638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:05.970667 32638 log.cc:824] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:05.972620 32638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2: No bootstrap required, opened a new log
I20260923 18:11:05.976227 32638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d3e998dd3b4733998dc369897ea9c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42361 } }
I20260923 18:11:05.976495 32638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:05.976547 32638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d3e998dd3b4733998dc369897ea9c2, State: Initialized, Role: FOLLOWER
I20260923 18:11:05.977028 32638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d3e998dd3b4733998dc369897ea9c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42361 } }
I20260923 18:11:05.977232 32638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:05.977311 32638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:05.977424 32638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:05.978434 32638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d3e998dd3b4733998dc369897ea9c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42361 } }
I20260923 18:11:05.978909 32638 leader_election.cc:302] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10d3e998dd3b4733998dc369897ea9c2; no voters:
I20260923 18:11:05.979285 32638 leader_election.cc:288] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:05.979398 32643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:05.979609 32643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [term 1 LEADER]: Becoming Leader. State: Replica: 10d3e998dd3b4733998dc369897ea9c2, State: Running, Role: LEADER
I20260923 18:11:05.979924 32643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d3e998dd3b4733998dc369897ea9c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42361 } }
I20260923 18:11:05.980561 32638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:05.981838 32645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10d3e998dd3b4733998dc369897ea9c2. Latest consensus state: current_term: 1 leader_uuid: "10d3e998dd3b4733998dc369897ea9c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d3e998dd3b4733998dc369897ea9c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42361 } } }
I20260923 18:11:05.981962 32645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:05.982197 32644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10d3e998dd3b4733998dc369897ea9c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d3e998dd3b4733998dc369897ea9c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42361 } } }
I20260923 18:11:05.982303 32644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:05.982379 32652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:05.985694 32652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:05.991514 32652 catalog_manager.cc:1347] Generated new cluster ID: 98009b5f58ba4632aa5f008358e47b81
I20260923 18:11:05.991600 32652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:06.012831 32652 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:06.013228 32652 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:06.022045 32652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2: Generated new TSK 0
I20260923 18:11:06.022799 32652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:11:06.031844 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42361
--builtin_ntp_servers=127.27.75.212:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 18:11:06.163619 32662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:06.163900 32662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:06.163939 32662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:06.168062 32662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:06.168129 32662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:06.168190 32662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:06.173005 32662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:06.174116 32662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:06.175457 32662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:11:06.185425 32662 server_base.cc:1034] running on GCE node
W20260923 18:11:06.185374 32668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:06.185369 32667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:06.185392 32670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:06.186062 32662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:06.186694 32662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:06.187870 32662 hybrid_clock.cc:648] HybridClock initialized: now 1790187066187835 us; error 47 us; skew 500 ppm
I20260923 18:11:06.189788 32662 webserver.cc:460] Webserver started at http://127.27.75.193:44949/ using document root <none> and password file <none>
I20260923 18:11:06.190348 32662 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:06.190414 32662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:06.190601 32662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:06.192664 32662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "483623a90634472f898caa1a6150f908"
format_stamp: "Formatted at 2026-09-23 18:11:06 on dist-test-slave-wsx5"
server_key: "08e1b16c81a25fa9f5dc7ec1c9cb2bc2"
server_key_iv: "109169c85e2ee9b5914ed1ecfcae6510"
server_key_version: "encryptionkey@0"
I20260923 18:11:06.193284 32662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "483623a90634472f898caa1a6150f908"
format_stamp: "Formatted at 2026-09-23 18:11:06 on dist-test-slave-wsx5"
server_key: "08e1b16c81a25fa9f5dc7ec1c9cb2bc2"
server_key_iv: "109169c85e2ee9b5914ed1ecfcae6510"
server_key_version: "encryptionkey@0"
I20260923 18:11:06.197005 32662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:06.199559 32676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:06.200793 32662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:11:06.200898 32662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "483623a90634472f898caa1a6150f908"
format_stamp: "Formatted at 2026-09-23 18:11:06 on dist-test-slave-wsx5"
server_key: "08e1b16c81a25fa9f5dc7ec1c9cb2bc2"
server_key_iv: "109169c85e2ee9b5914ed1ecfcae6510"
server_key_version: "encryptionkey@0"
I20260923 18:11:06.201018 32662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:06.205509 32662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:06.206107 32662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:06.206272 32662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:06.206794 32662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:06.207686 32662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:06.207733 32662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:06.207793 32662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:06.207818 32662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:06.216082 32662 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36363
I20260923 18:11:06.216111 321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36363 every 8 connection(s)
I20260923 18:11:06.217051 32662 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:06.218676 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 32662
I20260923 18:11:06.218796 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:11:06.219070 27951 external_mini_cluster.cc:1428] Setting key 22cb9b46ab887583dff654ebe3e101e8
I20260923 18:11:06.221213 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42361
--builtin_ntp_servers=127.27.75.212:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 18:11:06.223739 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.218973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:41165 (local address 127.27.75.254:42361)
0923 18:11:06.219165 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:06.219170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:06.220436 (+ 1266us) server_negotiation.cc:407] Connection header received
0923 18:11:06.222004 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:06.222007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:06.222051 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:06.222123 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:06.223285 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:06.223288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:06.223291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:06.223305 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:06.223326 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:06.223334 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:06.223419 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:06.223546 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 18:11:06.223593 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 18:11:06.224610 324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.219439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42361 (local address 127.27.75.193:41165)
0923 18:11:06.220280 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:06.220339 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:06.221786 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:06.222309 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:06.222316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:06.222414 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:06.222985 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:06.222987 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:06.223029 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:06.223112 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:06.223404 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:06.223412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:06.223417 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 18:11:06.223582 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 18:11:06.223836 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":144,"threads_started":1}
I20260923 18:11:06.225957 322 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42361
I20260923 18:11:06.226181 322 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:06.226590 322 heartbeater.cc:511] Master 127.27.75.254:42361 requested a full tablet report, sending...
I20260923 18:11:06.228161 32602 ts_manager.cc:194] Registered new tserver with Master: 483623a90634472f898caa1a6150f908 (127.27.75.193:36363)
W20260923 18:11:06.334200 325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:06.334481 325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:06.334538 325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:06.338563 325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:06.338625 325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:06.338706 325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:06.343573 325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:06.344681 325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:06.346088 325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:06.356019 331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:06.356019 330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:06.356093 333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:06.356281 325 server_base.cc:1034] running on GCE node
I20260923 18:11:06.356715 325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:06.357465 325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:06.358685 325 hybrid_clock.cc:648] HybridClock initialized: now 1790187066358661 us; error 44 us; skew 500 ppm
I20260923 18:11:06.360623 325 webserver.cc:460] Webserver started at http://127.27.75.194:39743/ using document root <none> and password file <none>
I20260923 18:11:06.361251 325 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:06.361340 325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:06.361552 325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:06.363633 325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "c708e3d1b0414edc8640d41d54fb3f2b"
format_stamp: "Formatted at 2026-09-23 18:11:06 on dist-test-slave-wsx5"
server_key: "655b9c0147e85d587f8cbaa62e72f755"
server_key_iv: "ff421681cd1319461537dde3df45bcd7"
server_key_version: "encryptionkey@0"
I20260923 18:11:06.364197 325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "c708e3d1b0414edc8640d41d54fb3f2b"
format_stamp: "Formatted at 2026-09-23 18:11:06 on dist-test-slave-wsx5"
server_key: "655b9c0147e85d587f8cbaa62e72f755"
server_key_iv: "ff421681cd1319461537dde3df45bcd7"
server_key_version: "encryptionkey@0"
I20260923 18:11:06.368445 325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:11:06.371286 339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:06.372756 325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:11:06.372900 325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "c708e3d1b0414edc8640d41d54fb3f2b"
format_stamp: "Formatted at 2026-09-23 18:11:06 on dist-test-slave-wsx5"
server_key: "655b9c0147e85d587f8cbaa62e72f755"
server_key_iv: "ff421681cd1319461537dde3df45bcd7"
server_key_version: "encryptionkey@0"
I20260923 18:11:06.373020 325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:06.377815 325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:06.378456 325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:06.378618 325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:06.379158 325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:06.380079 325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:06.380125 325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:06.380177 325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:06.380204 325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:06.388563 325 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:39397
I20260923 18:11:06.388577 452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:39397 every 8 connection(s)
I20260923 18:11:06.389568 325 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:06.395702 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.391418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:37937 (local address 127.27.75.254:42361)
0923 18:11:06.391569 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:06.391573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:06.392661 (+ 1088us) server_negotiation.cc:407] Connection header received
0923 18:11:06.394070 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:06.394073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:06.394114 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:06.394191 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:06.395296 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:06.395299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:06.395301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:06.395319 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:06.395341 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:06.395349 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:06.395434 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:06.395532 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 18:11:06.395572 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 18:11:06.396554 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.391797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42361 (local address 127.27.75.194:37937)
0923 18:11:06.392509 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:06.392555 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:06.393804 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:06.394366 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:06.394374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:06.394467 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:06.395020 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:06.395022 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:06.395065 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:06.395137 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:06.395420 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:06.395424 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:06.395428 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 18:11:06.395555 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 18:11:06.395756 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":112,"threads_started":1}
I20260923 18:11:06.397413 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 325
I20260923 18:11:06.397538 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:11:06.397831 27951 external_mini_cluster.cc:1428] Setting key 4f71b62b6dc2777255a6908c0458dd7f
I20260923 18:11:06.397837 453 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42361
I20260923 18:11:06.398077 453 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:06.398581 453 heartbeater.cc:511] Master 127.27.75.254:42361 requested a full tablet report, sending...
I20260923 18:11:06.399727 32602 ts_manager.cc:194] Registered new tserver with Master: c708e3d1b0414edc8640d41d54fb3f2b (127.27.75.194:39397)
I20260923 18:11:06.400007 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42361
--builtin_ntp_servers=127.27.75.212:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 18:11:06.511572 456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:06.511857 456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:06.511901 456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:06.516008 456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:06.516069 456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:06.516127 456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:06.520946 456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:06.522114 456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:06.523550 456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:06.533149 462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:06.533162 461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:06.533167 464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:06.533340 456 server_base.cc:1034] running on GCE node
I20260923 18:11:06.533799 456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:06.534420 456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:06.535640 456 hybrid_clock.cc:648] HybridClock initialized: now 1790187066535608 us; error 44 us; skew 500 ppm
I20260923 18:11:06.537490 456 webserver.cc:460] Webserver started at http://127.27.75.195:34341/ using document root <none> and password file <none>
I20260923 18:11:06.538023 456 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:06.538080 456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:06.538254 456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:06.540218 456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf"
format_stamp: "Formatted at 2026-09-23 18:11:06 on dist-test-slave-wsx5"
server_key: "0fcb1a698bb769055f6b809c46676e4c"
server_key_iv: "98e3fa8f9c639a19fd29671bc2f2df4a"
server_key_version: "encryptionkey@0"
I20260923 18:11:06.540732 456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf"
format_stamp: "Formatted at 2026-09-23 18:11:06 on dist-test-slave-wsx5"
server_key: "0fcb1a698bb769055f6b809c46676e4c"
server_key_iv: "98e3fa8f9c639a19fd29671bc2f2df4a"
server_key_version: "encryptionkey@0"
I20260923 18:11:06.544335 456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 18:11:06.546924 470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:06.548362 456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:11:06.548460 456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf"
format_stamp: "Formatted at 2026-09-23 18:11:06 on dist-test-slave-wsx5"
server_key: "0fcb1a698bb769055f6b809c46676e4c"
server_key_iv: "98e3fa8f9c639a19fd29671bc2f2df4a"
server_key_version: "encryptionkey@0"
I20260923 18:11:06.548568 456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:06.552940 456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:06.553565 456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:06.553720 456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:06.554221 456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:06.555100 456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:06.555145 456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:06.555184 456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:06.555212 456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:06.563783 456 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:43141
I20260923 18:11:06.563812 583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:43141 every 8 connection(s)
I20260923 18:11:06.564790 456 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:06.566417 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 456
I20260923 18:11:06.566543 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:11:06.566814 27951 external_mini_cluster.cc:1428] Setting key 25e13043a19d432f7541aab66c4d4466
I20260923 18:11:06.571023 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.568205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:55223 (local address 127.27.75.254:42361)
0923 18:11:06.568312 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:06.568316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:06.568330 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:06.569442 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:06.569445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:06.569482 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:06.569553 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:06.570596 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:06.570599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:06.570602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:06.570615 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:06.570635 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:06.570643 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:06.570723 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:06.570847 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 18:11:06.570890 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 18:11:06.571924 586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.567165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42361 (local address 127.27.75.195:55223)
0923 18:11:06.567981 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:06.568036 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:06.569228 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:06.569693 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:06.569700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:06.569787 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:06.570340 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:06.570342 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:06.570383 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:06.570452 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:06.570712 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:06.570720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:06.570726 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:11:06.570889 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 18:11:06.571133 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":122,"threads_started":1}
I20260923 18:11:06.572999 584 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42361
I20260923 18:11:06.573253 584 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:06.573724 584 heartbeater.cc:511] Master 127.27.75.254:42361 requested a full tablet report, sending...
I20260923 18:11:06.574714 32602 ts_manager.cc:194] Registered new tserver with Master: d8ecdc6e230646edbf21d4a3c23ea1cf (127.27.75.195:43141)
I20260923 18:11:06.580905 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:11:06.584386 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.582940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.27.75.254:42361)
0923 18:11:06.583086 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:06.583090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:06.583200 (+ 110us) server_negotiation.cc:407] Connection header received
0923 18:11:06.583325 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:06.583327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:06.583360 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:06.583427 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:06.584062 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:06.584065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:06.584067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:06.584077 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:06.584094 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:06.584102 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:06.584169 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:06.584226 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 18:11:06.584259 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:11:06.587196 32602 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:11:06.590195 32602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:11:06.602006 598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.598515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:43141 (local address 127.0.0.1:52588)
0923 18:11:06.599583 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:06.599634 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:06.599724 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:06.600195 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:06.600201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:06.600283 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:06.600811 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:06.600813 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:06.600834 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:06.600902 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:06.601650 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:06.601660 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:06.601666 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:11:06.601789 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 18:11:06.601859 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"spinlock_wait_cycles":1082752,"thread_start_us":121,"threads_started":1}
I20260923 18:11:06.602065 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.598740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36363 (local address 127.0.0.1:38840)
0923 18:11:06.599822 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:06.599832 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:06.599877 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:06.600872 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:06.600875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:06.600887 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:06.601442 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:06.601443 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:06.601452 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:06.601477 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:06.601847 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:06.601850 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:06.601853 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 18:11:06.601920 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 18:11:06.601951 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"spinlock_wait_cycles":1620224,"thread_start_us":90,"threads_started":1}
I20260923 18:11:06.602036 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.599145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.27.75.195:43141)
0923 18:11:06.599757 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:06.599767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:06.599791 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:11:06.599843 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:06.599849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:06.600045 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:06.600206 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:06.601416 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:06.601422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:06.601436 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:06.601447 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:06.601493 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:06.601530 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:06.601662 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:06.601802 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 18:11:06.601881 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":124,"threads_started":1}
I20260923 18:11:06.602700 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.598380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:39397 (local address 127.0.0.1:32788)
0923 18:11:06.599584 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:06.599637 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:06.599735 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:06.600811 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:06.600815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:06.600829 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:06.601382 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:06.601385 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:06.601396 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:06.601420 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:06.602480 (+ 1060us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:06.602482 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:06.602485 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 18:11:06.602550 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 18:11:06.602582 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"spinlock_wait_cycles":384,"thread_start_us":70,"threads_started":1}
I20260923 18:11:06.602782 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.598502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.27.75.194:39397)
0923 18:11:06.599434 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:06.599935 (+ 501us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:06.599956 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:11:06.599999 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:06.600003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:06.600141 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:06.600260 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:06.601560 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:06.601567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:06.601578 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:06.601591 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:06.601634 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:06.602346 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:06.602443 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:06.602581 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 18:11:06.602644 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":126,"threads_started":1}
I20260923 18:11:06.603497 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.599209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.27.75.193:36363)
0923 18:11:06.599580 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:06.599591 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:06.599906 (+ 315us) server_negotiation.cc:407] Connection header received
0923 18:11:06.599957 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:06.599961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:06.600090 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:06.600202 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:06.601595 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:06.601601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:06.601611 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:06.601623 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:06.601665 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:06.601716 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:06.601807 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:06.602655 (+ 848us) server_negotiation.cc:299] Negotiation successful
0923 18:11:06.602734 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":91,"threads_started":1}
I20260923 18:11:06.605340 387 tablet_service.cc:1467] Processing CreateTablet for tablet 6561e17212ab461eb32c5db3484721ef (DEFAULT_TABLE table=test-table [id=d00678a388d84db490bfa0bd9869e49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:06.605340 518 tablet_service.cc:1467] Processing CreateTablet for tablet 6561e17212ab461eb32c5db3484721ef (DEFAULT_TABLE table=test-table [id=d00678a388d84db490bfa0bd9869e49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:06.605700 32724 tablet_service.cc:1467] Processing CreateTablet for tablet 6561e17212ab461eb32c5db3484721ef (DEFAULT_TABLE table=test-table [id=d00678a388d84db490bfa0bd9869e49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:06.606472 518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6561e17212ab461eb32c5db3484721ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:06.606472 387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6561e17212ab461eb32c5db3484721ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:06.606572 32724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6561e17212ab461eb32c5db3484721ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:06.612582 601 tablet_bootstrap.cc:492] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908: Bootstrap starting.
I20260923 18:11:06.613306 602 tablet_bootstrap.cc:492] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: Bootstrap starting.
I20260923 18:11:06.614657 603 tablet_bootstrap.cc:492] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: Bootstrap starting.
I20260923 18:11:06.615650 602 tablet_bootstrap.cc:654] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:06.615693 601 tablet_bootstrap.cc:654] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:06.616389 601 log.cc:824] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:06.616456 602 log.cc:824] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:06.616616 603 tablet_bootstrap.cc:654] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:06.617373 603 log.cc:824] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:06.618541 601 tablet_bootstrap.cc:492] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908: No bootstrap required, opened a new log
I20260923 18:11:06.618742 601 ts_tablet_manager.cc:1397] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:11:06.618940 602 tablet_bootstrap.cc:492] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: No bootstrap required, opened a new log
I20260923 18:11:06.619016 603 tablet_bootstrap.cc:492] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: No bootstrap required, opened a new log
I20260923 18:11:06.619102 602 ts_tablet_manager.cc:1397] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:11:06.619180 603 ts_tablet_manager.cc:1397] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 18:11:06.622537 603 raft_consensus.cc:348] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 39397 } } peers { permanent_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43141 } } peers { permanent_uuid: "483623a90634472f898caa1a6150f908" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36363 } }
I20260923 18:11:06.622774 603 raft_consensus.cc:374] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:06.622834 603 raft_consensus.cc:729] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c708e3d1b0414edc8640d41d54fb3f2b, State: Initialized, Role: FOLLOWER
I20260923 18:11:06.623243 603 consensus_queue.cc:260] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 39397 } } peers { permanent_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43141 } } peers { permanent_uuid: "483623a90634472f898caa1a6150f908" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36363 } }
I20260923 18:11:06.623224 601 raft_consensus.cc:348] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 39397 } } peers { permanent_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43141 } } peers { permanent_uuid: "483623a90634472f898caa1a6150f908" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36363 } }
I20260923 18:11:06.623224 602 raft_consensus.cc:348] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 39397 } } peers { permanent_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43141 } } peers { permanent_uuid: "483623a90634472f898caa1a6150f908" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36363 } }
I20260923 18:11:06.623471 601 raft_consensus.cc:374] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:06.623471 602 raft_consensus.cc:374] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:06.623528 602 raft_consensus.cc:729] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ecdc6e230646edbf21d4a3c23ea1cf, State: Initialized, Role: FOLLOWER
I20260923 18:11:06.623528 601 raft_consensus.cc:729] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 483623a90634472f898caa1a6150f908, State: Initialized, Role: FOLLOWER
I20260923 18:11:06.623965 602 consensus_queue.cc:260] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 39397 } } peers { permanent_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43141 } } peers { permanent_uuid: "483623a90634472f898caa1a6150f908" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36363 } }
I20260923 18:11:06.623965 601 consensus_queue.cc:260] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 39397 } } peers { permanent_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43141 } } peers { permanent_uuid: "483623a90634472f898caa1a6150f908" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36363 } }
I20260923 18:11:06.624168 603 ts_tablet_manager.cc:1428] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 18:11:06.624506 453 heartbeater.cc:503] Master 127.27.75.254:42361 was elected leader, sending a full tablet report...
I20260923 18:11:06.624886 602 ts_tablet_manager.cc:1428] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:11:06.624886 601 ts_tablet_manager.cc:1428] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:11:06.625239 322 heartbeater.cc:503] Master 127.27.75.254:42361 was elected leader, sending a full tablet report...
I20260923 18:11:06.625239 584 heartbeater.cc:503] Master 127.27.75.254:42361 was elected leader, sending a full tablet report...
W20260923 18:11:06.640586 454 tablet.cc:2369] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:11:06.718312 323 tablet.cc:2369] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:11:06.815826 585 tablet.cc:2369] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:11:06.868953 609 raft_consensus.cc:482] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:11:06.869197 609 raft_consensus.cc:504] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 39397 } } peers { permanent_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43141 } } peers { permanent_uuid: "483623a90634472f898caa1a6150f908" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36363 } }
I20260923 18:11:06.870411 609 leader_election.cc:288] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c708e3d1b0414edc8640d41d54fb3f2b (127.27.75.194:39397), d8ecdc6e230646edbf21d4a3c23ea1cf (127.27.75.195:43141)
I20260923 18:11:06.872629 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.870803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:37903 (local address 127.27.75.195:43141)
0923 18:11:06.870937 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:06.870942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:06.871179 (+ 237us) server_negotiation.cc:407] Connection header received
0923 18:11:06.871293 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:06.871296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:06.871337 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:06.871399 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:06.872247 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:06.872251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:06.872254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:06.872267 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:06.872286 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:06.872295 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:06.872367 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:06.872465 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 18:11:06.872503 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 18:11:06.872629 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.870820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:43669 (local address 127.27.75.194:39397)
0923 18:11:06.870937 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:06.870942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:06.871212 (+ 270us) server_negotiation.cc:407] Connection header received
0923 18:11:06.871318 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:06.871321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:06.871353 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:06.871432 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:06.872271 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:06.872274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:06.872276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:06.872286 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:06.872304 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:06.872311 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:06.872392 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:06.872460 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 18:11:06.872496 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 18:11:06.872644 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.870692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:43141 (local address 127.27.75.193:37903)
0923 18:11:06.871076 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:06.871088 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:06.871186 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:06.871442 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:06.871444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:06.871460 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:06.872043 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:06.872046 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:06.872058 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:06.872090 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:06.872367 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:06.872371 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:06.872374 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 18:11:06.872452 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 18:11:06.872493 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":104,"threads_started":1}
I20260923 18:11:06.872654 611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.870694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:39397 (local address 127.27.75.193:43669)
0923 18:11:06.871111 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:06.871123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:06.871227 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:06.871467 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:06.871470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:06.871484 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:06.872099 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:06.872100 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:06.872110 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:06.872142 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:06.872387 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:06.872389 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:06.872392 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 18:11:06.872464 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:11:06.872500 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":189,"thread_start_us":64,"threads_started":1}
I20260923 18:11:06.873289 538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6561e17212ab461eb32c5db3484721ef" candidate_uuid: "483623a90634472f898caa1a6150f908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" is_pre_election: true
I20260923 18:11:06.873507 407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6561e17212ab461eb32c5db3484721ef" candidate_uuid: "483623a90634472f898caa1a6150f908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" is_pre_election: true
I20260923 18:11:06.873641 538 raft_consensus.cc:2454] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 483623a90634472f898caa1a6150f908 in term 0.
I20260923 18:11:06.873785 407 raft_consensus.cc:2454] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 483623a90634472f898caa1a6150f908 in term 0.
I20260923 18:11:06.874157 32677 leader_election.cc:302] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 483623a90634472f898caa1a6150f908, c708e3d1b0414edc8640d41d54fb3f2b, d8ecdc6e230646edbf21d4a3c23ea1cf; no voters:
I20260923 18:11:06.874455 609 raft_consensus.cc:2781] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:11:06.874524 609 raft_consensus.cc:482] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:11:06.874557 609 raft_consensus.cc:3037] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:06.875550 609 raft_consensus.cc:504] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 39397 } } peers { permanent_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43141 } } peers { permanent_uuid: "483623a90634472f898caa1a6150f908" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36363 } }
I20260923 18:11:06.875890 609 leader_election.cc:288] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [CANDIDATE]: Term 1 election: Requested vote from peers c708e3d1b0414edc8640d41d54fb3f2b (127.27.75.194:39397), d8ecdc6e230646edbf21d4a3c23ea1cf (127.27.75.195:43141)
I20260923 18:11:06.876237 407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6561e17212ab461eb32c5db3484721ef" candidate_uuid: "483623a90634472f898caa1a6150f908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c708e3d1b0414edc8640d41d54fb3f2b"
I20260923 18:11:06.876279 538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6561e17212ab461eb32c5db3484721ef" candidate_uuid: "483623a90634472f898caa1a6150f908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf"
I20260923 18:11:06.876343 407 raft_consensus.cc:3037] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:06.876437 538 raft_consensus.cc:3037] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:06.877447 407 raft_consensus.cc:2454] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 483623a90634472f898caa1a6150f908 in term 1.
I20260923 18:11:06.877696 538 raft_consensus.cc:2454] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 483623a90634472f898caa1a6150f908 in term 1.
I20260923 18:11:06.877715 32680 leader_election.cc:302] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 483623a90634472f898caa1a6150f908, c708e3d1b0414edc8640d41d54fb3f2b; no voters:
I20260923 18:11:06.877887 609 raft_consensus.cc:2781] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:06.878099 609 raft_consensus.cc:686] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [term 1 LEADER]: Becoming Leader. State: Replica: 483623a90634472f898caa1a6150f908, State: Running, Role: LEADER
I20260923 18:11:06.878413 609 consensus_queue.cc:237] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 39397 } } peers { permanent_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43141 } } peers { permanent_uuid: "483623a90634472f898caa1a6150f908" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36363 } }
I20260923 18:11:06.881820 32601 catalog_manager.cc:5495] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 reported cstate change: term changed from 0 to 1, leader changed from <none> to 483623a90634472f898caa1a6150f908 (127.27.75.193). New cstate: current_term: 1 leader_uuid: "483623a90634472f898caa1a6150f908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "483623a90634472f898caa1a6150f908" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36363 } health_report { overall_health: HEALTHY } } }
I20260923 18:11:06.901798 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.900522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.27.75.193:36363)
0923 18:11:06.900634 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:06.900637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:06.900650 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:06.900723 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:06.900725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:06.900816 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:06.900893 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:06.901473 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:06.901476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:06.901478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:06.901489 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:06.901507 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:06.901515 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:06.901580 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:06.901630 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 18:11:06.901662 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 18:11:06.907428 407 raft_consensus.cc:1261] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b [term 1 FOLLOWER]: Refusing update from remote peer 483623a90634472f898caa1a6150f908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:06.907428 538 raft_consensus.cc:1261] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf [term 1 FOLLOWER]: Refusing update from remote peer 483623a90634472f898caa1a6150f908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:06.908058 612 consensus_queue.cc:1035] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ecdc6e230646edbf21d4a3c23ea1cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:06.908237 609 consensus_queue.cc:1035] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c708e3d1b0414edc8640d41d54fb3f2b" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:06.917433 616 mvcc.cc:205] Tried to move back new op lower bound from 7332606226046070784 to 7332606225936343040. Current Snapshot: MvccSnapshot[applied={T|T < 7332606226046070784}]
I20260923 18:11:06.927294 32602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:06.927485 32602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:06.929955 32602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10d3e998dd3b4733998dc369897ea9c2: Sending DeleteTablet for 3 replicas of tablet 6561e17212ab461eb32c5db3484721ef
I20260923 18:11:06.930757 387 tablet_service.cc:1514] Processing DeleteTablet for tablet 6561e17212ab461eb32c5db3484721ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:06 UTC) from {username='slave'} at 127.0.0.1:32788
I20260923 18:11:06.930748 518 tablet_service.cc:1514] Processing DeleteTablet for tablet 6561e17212ab461eb32c5db3484721ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:06 UTC) from {username='slave'} at 127.0.0.1:52588
I20260923 18:11:06.930896 32724 tablet_service.cc:1514] Processing DeleteTablet for tablet 6561e17212ab461eb32c5db3484721ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:06 UTC) from {username='slave'} at 127.0.0.1:38840
I20260923 18:11:06.931263 624 tablet_replica.cc:330] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: stopping tablet replica
I20260923 18:11:06.931489 626 tablet_replica.cc:330] T 6561e17212ab461eb32c5db3484721ef P 483623a90634472f898caa1a6150f908: stopping tablet replica
I20260923 18:11:06.931599 624 raft_consensus.cc:2229] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:06.931720 625 tablet_replica.cc:330] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: stopping tablet replica
I20260923 18:11:06.931943 624 raft_consensus.cc:2258] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:06.932099 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 32662
I20260923 18:11:06.933565 624 ts_tablet_manager.cc:1905] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:06.934331 625 raft_consensus.cc:2229] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:06.934636 625 raft_consensus.cc:2258] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:06.935999 625 ts_tablet_manager.cc:1905] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:06.936332 624 ts_tablet_manager.cc:1918] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:06.936429 624 log.cc:1196] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/6561e17212ab461eb32c5db3484721ef
I20260923 18:11:06.936705 624 ts_tablet_manager.cc:1939] T 6561e17212ab461eb32c5db3484721ef P d8ecdc6e230646edbf21d4a3c23ea1cf: Deleting consensus metadata
I20260923 18:11:06.938097 32586 catalog_manager.cc:4841] TS d8ecdc6e230646edbf21d4a3c23ea1cf (127.27.75.195:43141): tablet 6561e17212ab461eb32c5db3484721ef (table test-table [id=d00678a388d84db490bfa0bd9869e49c]) successfully deleted
I20260923 18:11:06.938300 625 ts_tablet_manager.cc:1918] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:06.938398 625 log.cc:1196] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/6561e17212ab461eb32c5db3484721ef
I20260923 18:11:06.938644 625 ts_tablet_manager.cc:1939] T 6561e17212ab461eb32c5db3484721ef P c708e3d1b0414edc8640d41d54fb3f2b: Deleting consensus metadata
W20260923 18:11:06.939298 32588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:36363 (error 108)
I20260923 18:11:06.939466 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 325
I20260923 18:11:06.941479 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.941085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36363 (local address 127.0.0.1:55606)
0923 18:11:06.941283 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:06.941386 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260923 18:11:06.941728 32588 catalog_manager.cc:4584] TS 483623a90634472f898caa1a6150f908 (127.27.75.193:36363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6561e17212ab461eb32c5db3484721ef: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36363: connect: Connection refused (error 111)
I20260923 18:11:06.944953 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 456
I20260923 18:11:06.946228 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:06.945189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:39397 (local address 127.0.0.1:38986)
0923 18:11:06.946069 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:06.946128 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.194:39397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":772}
I20260923 18:11:06.949688 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 32571
2026-09-23T18:11:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1185 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 18:11:06.969858 27951 mini_oidc.cc:152] Starting JWKS server
I20260923 18:11:06.971387 27951 webserver.cc:460] Webserver started at https://127.0.0.1:43397/ using document root <none> and password file <none>
I20260923 18:11:06.971526 27951 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 18:11:06.971755 27951 webserver.cc:460] Webserver started at http://127.0.0.1:38449/ using document root <none> and password file <none>
2026-09-23T18:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:06Z Disabled control of system clock
I20260923 18:11:06.980979 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38063
--webserver_interface=127.27.75.254
--webserver_port=0
--builtin_ntp_servers=127.27.75.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:38063
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43397/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/testchainca.cert with env {}
W20260923 18:11:07.093310 633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:07.093612 633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:07.093655 633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:07.097820 633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:07.097891 633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:07.097906 633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:07.097921 633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:07.097936 633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 18:11:07.097951 633 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 18:11:07.097963 633 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43397/jwks.json
I20260923 18:11:07.103607 633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:38063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43397/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:07.104671 633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:07.105711 633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:07.114588 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:07.114620 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:07.114758 633 server_base.cc:1034] running on GCE node
W20260923 18:11:07.114620 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:07.115464 633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:07.116647 633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:07.117815 633 hybrid_clock.cc:648] HybridClock initialized: now 1790187067117791 us; error 35 us; skew 500 ppm
I20260923 18:11:07.119733 633 webserver.cc:460] Webserver started at http://127.27.75.254:40885/ using document root <none> and password file <none>
I20260923 18:11:07.120282 633 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:07.120342 633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:07.120527 633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:07.122628 633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "3b8a5491d6474da2bea08899ef0ae8c2"
format_stamp: "Formatted at 2026-09-23 18:11:07 on dist-test-slave-wsx5"
server_key: "e42baec3fbb51738c4587f512ea7677b"
server_key_iv: "895bb4969878784f07f71df1e3c7e6e1"
server_key_version: "encryptionkey@0"
I20260923 18:11:07.123183 633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "3b8a5491d6474da2bea08899ef0ae8c2"
format_stamp: "Formatted at 2026-09-23 18:11:07 on dist-test-slave-wsx5"
server_key: "e42baec3fbb51738c4587f512ea7677b"
server_key_iv: "895bb4969878784f07f71df1e3c7e6e1"
server_key_version: "encryptionkey@0"
I20260923 18:11:07.126956 633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:11:07.129448 647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:07.130800 633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 18:11:07.130914 633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "3b8a5491d6474da2bea08899ef0ae8c2"
format_stamp: "Formatted at 2026-09-23 18:11:07 on dist-test-slave-wsx5"
server_key: "e42baec3fbb51738c4587f512ea7677b"
server_key_iv: "895bb4969878784f07f71df1e3c7e6e1"
server_key_version: "encryptionkey@0"
I20260923 18:11:07.131028 633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:07.157887 633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:07.158634 633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:07.158826 633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:07.165318 633 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:38063
I20260923 18:11:07.165325 699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:38063 every 8 connection(s)
I20260923 18:11:07.166254 633 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:07.167265 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 633
I20260923 18:11:07.167397 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:07.167735 27951 external_mini_cluster.cc:1428] Setting key ce0184e9d19f3d12ee72557b048d4d51
I20260923 18:11:07.170369 700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:07.176664 700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2: Bootstrap starting.
I20260923 18:11:07.176729 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:07.168973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.27.75.254:38063)
0923 18:11:07.169585 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:07.169600 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:07.169636 (+ 36us) server_negotiation.cc:407] Connection header received
0923 18:11:07.170277 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:07.170293 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:07.170713 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:07.171012 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:07.171429 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.172392 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.173615 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.174117 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.174243 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:07.175050 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:07.175055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:07.175067 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:07.175446 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:07.175500 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:07.175520 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:07.175592 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:07.175764 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 18:11:07.176026 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":137,"threads_started":1}
I20260923 18:11:07.179129 700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:07.179498 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:11:07.179896 700 log.cc:824] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:07.181877 700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2: No bootstrap required, opened a new log
I20260923 18:11:07.185364 700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8a5491d6474da2bea08899ef0ae8c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38063 } }
I20260923 18:11:07.185596 700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:07.185652 700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8a5491d6474da2bea08899ef0ae8c2, State: Initialized, Role: FOLLOWER
I20260923 18:11:07.186055 700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8a5491d6474da2bea08899ef0ae8c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38063 } }
I20260923 18:11:07.186183 700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:07.186256 700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:07.186352 700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:07.187186 700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8a5491d6474da2bea08899ef0ae8c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38063 } }
I20260923 18:11:07.187513 700 leader_election.cc:302] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b8a5491d6474da2bea08899ef0ae8c2; no voters:
I20260923 18:11:07.187808 700 leader_election.cc:288] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:07.187935 705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:07.188141 705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [term 1 LEADER]: Becoming Leader. State: Replica: 3b8a5491d6474da2bea08899ef0ae8c2, State: Running, Role: LEADER
I20260923 18:11:07.188414 705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8a5491d6474da2bea08899ef0ae8c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38063 } }
I20260923 18:11:07.188949 700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:07.189996 706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b8a5491d6474da2bea08899ef0ae8c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8a5491d6474da2bea08899ef0ae8c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38063 } } }
I20260923 18:11:07.190122 706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:07.190368 707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b8a5491d6474da2bea08899ef0ae8c2. Latest consensus state: current_term: 1 leader_uuid: "3b8a5491d6474da2bea08899ef0ae8c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8a5491d6474da2bea08899ef0ae8c2" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38063 } } }
I20260923 18:11:07.190444 707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:07.190482 714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:07.193391 714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:07.198819 714 catalog_manager.cc:1347] Generated new cluster ID: f38c81b90c574e069003d0095504ae3e
I20260923 18:11:07.198884 714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:07.211148 714 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:07.212193 714 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:07.223424 714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b8a5491d6474da2bea08899ef0ae8c2: Generated new TSK 0
I20260923 18:11:07.224128 714 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 18:11:07.257717 730 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 18:11:07.257826 730 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 18:11:07.258313 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:07.240587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.27.75.254:38063)
0923 18:11:07.240729 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:07.240733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:07.240931 (+ 198us) server_negotiation.cc:407] Connection header received
0923 18:11:07.241146 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:07.241149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:07.241208 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:07.241314 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0923 18:11:07.241543 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.242374 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.243274 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.243608 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.243789 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:07.244123 (+ 334us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 18:11:07.257891 (+ 13768us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 18:11:07.258172 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59354: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260923 18:11:07.258378 703 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59354: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 18:11:07.259442 729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:07.240479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38063 (local address 127.0.0.1:59354)
0923 18:11:07.240855 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:07.240868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:07.240976 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:07.241329 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:07.241332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:07.241339 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 18:11:07.241446 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:07.241451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.242501 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.242504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:07.243159 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:07.243169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.243785 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.243788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:07.243863 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:07.243912 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 18:11:07.258121 (+ 14209us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 18:11:07.258334 (+ 213us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.75.254:38063: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.75.254:38063: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 18:11:07.260234 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 633
2026-09-23T18:11:07Z chronyd exiting
I20260923 18:11:07.296911 27951 test_util.cc:175] -----------------------------------------------
I20260923 18:11:07.297027 27951 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790187042487232-27951-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (328 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 18:11:07.298288 27951 mini_oidc.cc:152] Starting JWKS server
I20260923 18:11:07.299113 27951 webserver.cc:460] Webserver started at https://127.0.0.1:42141/ using document root <none> and password file <none>
I20260923 18:11:07.299199 27951 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 18:11:07.299407 27951 webserver.cc:460] Webserver started at http://127.0.0.1:40601/ using document root <none> and password file <none>
2026-09-23T18:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:07Z Disabled control of system clock
I20260923 18:11:07.308876 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:46645
--webserver_interface=127.27.75.254
--webserver_port=0
--builtin_ntp_servers=127.27.75.212:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:46645
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42141/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/testchainca.cert with env {}
W20260923 18:11:07.418615 736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:07.418910 736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:07.418968 736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:07.423075 736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:07.423152 736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:07.423174 736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:07.423197 736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:07.423215 736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 18:11:07.423250 736 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 18:11:07.423276 736 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42141/jwks.json
I20260923 18:11:07.428928 736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:46645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:46645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42141/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:07.430155 736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:07.431188 736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:07.439891 742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:07.439882 741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:07.439886 744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:07.440037 736 server_base.cc:1034] running on GCE node
I20260923 18:11:07.440734 736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:07.441984 736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:07.443153 736 hybrid_clock.cc:648] HybridClock initialized: now 1790187067443131 us; error 33 us; skew 500 ppm
I20260923 18:11:07.445070 736 webserver.cc:460] Webserver started at http://127.27.75.254:33427/ using document root <none> and password file <none>
I20260923 18:11:07.445675 736 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:07.445740 736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:07.445924 736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:07.447942 736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "0b4db0aa3c8f46bcb4ac4c7aaa58e678"
format_stamp: "Formatted at 2026-09-23 18:11:07 on dist-test-slave-wsx5"
server_key: "4929937af20cac8b59e39d48c6c108fc"
server_key_iv: "68fa1c95cb4a99306407b25dd0b0a60d"
server_key_version: "encryptionkey@0"
I20260923 18:11:07.448475 736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "0b4db0aa3c8f46bcb4ac4c7aaa58e678"
format_stamp: "Formatted at 2026-09-23 18:11:07 on dist-test-slave-wsx5"
server_key: "4929937af20cac8b59e39d48c6c108fc"
server_key_iv: "68fa1c95cb4a99306407b25dd0b0a60d"
server_key_version: "encryptionkey@0"
I20260923 18:11:07.452215 736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:11:07.454834 750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:07.456080 736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:11:07.456229 736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "0b4db0aa3c8f46bcb4ac4c7aaa58e678"
format_stamp: "Formatted at 2026-09-23 18:11:07 on dist-test-slave-wsx5"
server_key: "4929937af20cac8b59e39d48c6c108fc"
server_key_iv: "68fa1c95cb4a99306407b25dd0b0a60d"
server_key_version: "encryptionkey@0"
I20260923 18:11:07.456333 736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:07.486914 736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:07.487643 736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:07.487834 736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:07.494781 736 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:46645
I20260923 18:11:07.494799 802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:46645 every 8 connection(s)
I20260923 18:11:07.495771 736 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:07.498974 803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:07.504956 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 736
I20260923 18:11:07.505084 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:07.505383 27951 external_mini_cluster.cc:1428] Setting key 6303b950d82686a173c9b762eceb22d6
I20260923 18:11:07.505388 803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678: Bootstrap starting.
I20260923 18:11:07.508313 803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:07.509292 803 log.cc:824] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:07.511650 803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678: No bootstrap required, opened a new log
I20260923 18:11:07.516059 806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:07.506844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.27.75.254:46645)
0923 18:11:07.507593 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:07.507613 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:07.507660 (+ 47us) server_negotiation.cc:407] Connection header received
0923 18:11:07.508554 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:07.508582 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:07.509179 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:07.509580 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:07.509985 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.511351 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.512266 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.512933 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.513114 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:07.513869 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:07.513875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:07.513893 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:07.514417 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:07.514487 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:07.514513 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:07.514614 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:07.514741 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 18:11:07.515054 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":133,"threads_started":1}
I20260923 18:11:07.516091 803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4db0aa3c8f46bcb4ac4c7aaa58e678" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46645 } }
I20260923 18:11:07.516391 803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:07.516438 803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4db0aa3c8f46bcb4ac4c7aaa58e678, State: Initialized, Role: FOLLOWER
I20260923 18:11:07.516841 803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4db0aa3c8f46bcb4ac4c7aaa58e678" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46645 } }
I20260923 18:11:07.516986 803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:07.517043 803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:07.517156 803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:07.518392 803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4db0aa3c8f46bcb4ac4c7aaa58e678" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46645 } }
I20260923 18:11:07.518769 803 leader_election.cc:302] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b4db0aa3c8f46bcb4ac4c7aaa58e678; no voters:
I20260923 18:11:07.519112 803 leader_election.cc:288] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:07.519223 808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:07.519452 808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [term 1 LEADER]: Becoming Leader. State: Replica: 0b4db0aa3c8f46bcb4ac4c7aaa58e678, State: Running, Role: LEADER
I20260923 18:11:07.519758 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:11:07.519886 808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4db0aa3c8f46bcb4ac4c7aaa58e678" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46645 } }
I20260923 18:11:07.520439 803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:07.522532 809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b4db0aa3c8f46bcb4ac4c7aaa58e678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4db0aa3c8f46bcb4ac4c7aaa58e678" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46645 } } }
I20260923 18:11:07.522676 809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:07.522876 809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b4db0aa3c8f46bcb4ac4c7aaa58e678. Latest consensus state: current_term: 1 leader_uuid: "0b4db0aa3c8f46bcb4ac4c7aaa58e678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4db0aa3c8f46bcb4ac4c7aaa58e678" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46645 } } }
I20260923 18:11:07.522949 809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678 [sys.catalog]: This master's current role is: LEADER
W20260923 18:11:07.526023 822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:11:07.526096 822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:11:07.526188 823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:07.526870 823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:07.532452 823 catalog_manager.cc:1347] Generated new cluster ID: fa6e1dc5296349bfaf9bfdb3c00a45bc
I20260923 18:11:07.532528 823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:07.541982 823 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:07.543292 823 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:07.552861 823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b4db0aa3c8f46bcb4ac4c7aaa58e678: Generated new TSK 0
I20260923 18:11:07.553807 823 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 18:11:07.578423 833 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 18:11:07.578503 833 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 18:11:07.579331 806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:07.562901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.27.75.254:46645)
0923 18:11:07.563038 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:07.563044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:07.563181 (+ 137us) server_negotiation.cc:407] Connection header received
0923 18:11:07.563362 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:07.563366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:07.563429 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:07.563556 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0923 18:11:07.563749 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.564599 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.565529 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.565883 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.565981 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:07.566271 (+ 290us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 18:11:07.578854 (+ 12583us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 18:11:07.579166 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37944: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260923 18:11:07.579407 806 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37944: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 18:11:07.579418 832 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:07.562802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:46645 (local address 127.0.0.1:37944)
0923 18:11:07.563100 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:07.563112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:07.563213 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:07.563549 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:07.563551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:07.563558 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 18:11:07.563656 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:07.563660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.564732 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.564736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:07.565420 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:07.565429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.566000 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.566003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:07.566066 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:07.566092 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 18:11:07.579078 (+ 12986us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 18:11:07.579265 (+ 187us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.75.254:46645: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
I20260923 18:11:07.580152 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 736
2026-09-23T18:11:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (320 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 18:11:07.618839 27951 mini_oidc.cc:152] Starting JWKS server
I20260923 18:11:07.619685 27951 webserver.cc:460] Webserver started at https://127.0.0.1:33017/ using document root <none> and password file <none>
I20260923 18:11:07.619788 27951 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 18:11:07.620025 27951 webserver.cc:460] Webserver started at http://127.0.0.1:40889/ using document root <none> and password file <none>
2026-09-23T18:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:07Z Disabled control of system clock
I20260923 18:11:07.628247 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39305
--webserver_interface=127.27.75.254
--webserver_port=0
--builtin_ntp_servers=127.27.75.212:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:39305
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33017/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 18:11:07.739008 839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:07.739306 839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:07.739346 839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:07.743660 839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:07.743741 839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:07.743755 839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:07.743772 839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:07.743786 839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 18:11:07.743801 839 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 18:11:07.743813 839 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33017/jwks.json
I20260923 18:11:07.748951 839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:39305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33017/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:07.750160 839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:07.751605 839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:07.760432 847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:07.760495 844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:07.760486 839 server_base.cc:1034] running on GCE node
W20260923 18:11:07.760411 845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:07.761307 839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:07.762457 839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:07.763609 839 hybrid_clock.cc:648] HybridClock initialized: now 1790187067763587 us; error 40 us; skew 500 ppm
I20260923 18:11:07.765609 839 webserver.cc:460] Webserver started at http://127.27.75.254:33973/ using document root <none> and password file <none>
I20260923 18:11:07.766201 839 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:07.766268 839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:07.766465 839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:07.768625 839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "1f36efe6f1aa46729b95dc7cb26c66bf"
format_stamp: "Formatted at 2026-09-23 18:11:07 on dist-test-slave-wsx5"
server_key: "b5e31a27519db06253bcbec23141464b"
server_key_iv: "65a7b5dbf3a6743f2274d09e614531e4"
server_key_version: "encryptionkey@0"
I20260923 18:11:07.769359 839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "1f36efe6f1aa46729b95dc7cb26c66bf"
format_stamp: "Formatted at 2026-09-23 18:11:07 on dist-test-slave-wsx5"
server_key: "b5e31a27519db06253bcbec23141464b"
server_key_iv: "65a7b5dbf3a6743f2274d09e614531e4"
server_key_version: "encryptionkey@0"
I20260923 18:11:07.773700 839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:11:07.776465 853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:07.777951 839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:07.778088 839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "1f36efe6f1aa46729b95dc7cb26c66bf"
format_stamp: "Formatted at 2026-09-23 18:11:07 on dist-test-slave-wsx5"
server_key: "b5e31a27519db06253bcbec23141464b"
server_key_iv: "65a7b5dbf3a6743f2274d09e614531e4"
server_key_version: "encryptionkey@0"
I20260923 18:11:07.778215 839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:07.792514 839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:07.793349 839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:07.793567 839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:07.800154 839 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:39305
I20260923 18:11:07.800165 905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:39305 every 8 connection(s)
I20260923 18:11:07.801084 839 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:07.803792 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 839
I20260923 18:11:07.803900 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:07.804154 27951 external_mini_cluster.cc:1428] Setting key 9fc9300d7bb79a48799694e81b6b6c61
I20260923 18:11:07.804247 906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:07.810853 906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf: Bootstrap starting.
I20260923 18:11:07.813196 906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:07.813371 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:07.805539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.27.75.254:39305)
0923 18:11:07.806301 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:07.806317 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:07.806357 (+ 40us) server_negotiation.cc:407] Connection header received
0923 18:11:07.806991 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:07.807019 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:07.807460 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:07.807803 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:07.808262 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.809290 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.810191 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.810662 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.810819 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:07.811644 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:07.811648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:07.811661 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:07.812038 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:07.812097 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:07.812116 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:07.812213 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:07.812336 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 18:11:07.812592 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":138,"threads_started":1}
I20260923 18:11:07.814074 906 log.cc:824] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:07.816201 906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf: No bootstrap required, opened a new log
I20260923 18:11:07.816474 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:11:07.819802 906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36efe6f1aa46729b95dc7cb26c66bf" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39305 } }
I20260923 18:11:07.820019 906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:07.820070 906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f36efe6f1aa46729b95dc7cb26c66bf, State: Initialized, Role: FOLLOWER
I20260923 18:11:07.820492 906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36efe6f1aa46729b95dc7cb26c66bf" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39305 } }
I20260923 18:11:07.820626 906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:07.820689 906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:07.820782 906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:07.821727 906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36efe6f1aa46729b95dc7cb26c66bf" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39305 } }
I20260923 18:11:07.822041 906 leader_election.cc:302] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f36efe6f1aa46729b95dc7cb26c66bf; no voters:
I20260923 18:11:07.822333 906 leader_election.cc:288] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:07.822465 911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:07.822708 911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [term 1 LEADER]: Becoming Leader. State: Replica: 1f36efe6f1aa46729b95dc7cb26c66bf, State: Running, Role: LEADER
I20260923 18:11:07.823040 911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36efe6f1aa46729b95dc7cb26c66bf" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39305 } }
I20260923 18:11:07.823559 906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:07.824666 913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f36efe6f1aa46729b95dc7cb26c66bf. Latest consensus state: current_term: 1 leader_uuid: "1f36efe6f1aa46729b95dc7cb26c66bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36efe6f1aa46729b95dc7cb26c66bf" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39305 } } }
I20260923 18:11:07.824800 913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:07.824990 912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f36efe6f1aa46729b95dc7cb26c66bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f36efe6f1aa46729b95dc7cb26c66bf" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39305 } } }
I20260923 18:11:07.825157 912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:07.825232 920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:07.827970 920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:07.833319 920 catalog_manager.cc:1347] Generated new cluster ID: 0b88bf8e9fdf4b07a3c791e0ec66992d
I20260923 18:11:07.833408 920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:07.855631 920 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:07.856592 920 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:07.865280 920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f36efe6f1aa46729b95dc7cb26c66bf: Generated new TSK 0
I20260923 18:11:07.866014 920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:11:07.887576 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:07.881335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.27.75.254:39305)
0923 18:11:07.881487 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:07.881492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:07.881751 (+ 259us) server_negotiation.cc:407] Connection header received
0923 18:11:07.881855 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:07.881858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:07.881921 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:07.882044 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0923 18:11:07.882372 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.883207 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.884599 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.884843 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.884936 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:07.885340 (+ 404us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 18:11:07.887217 (+ 1877us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 18:11:07.887451 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39716: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 18:11:07.887641 909 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39716: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 18:11:07.890478 935 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:07.881286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39305 (local address 127.0.0.1:39716)
0923 18:11:07.881602 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:07.881617 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:07.881725 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:07.882054 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:07.882057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:07.882065 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0923 18:11:07.882240 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:07.882262 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.883405 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.883409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:07.884443 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:07.884456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:07.884971 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:07.884975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:07.885068 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:07.885100 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 18:11:07.887377 (+ 2277us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 18:11:07.890330 (+ 2953us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.75.254:39305: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":2739,"thread_start_us":122,"threads_started":1}
I20260923 18:11:07.890905 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 839
E20260923 18:11:07.899657 936 webserver.cc:533] Webserver: sslize failed:
E20260923 18:11:07.899747 936 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-23T18:11:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (310 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T18:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:07Z Disabled control of system clock
I20260923 18:11:07.937429 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42841
--webserver_interface=127.27.75.254
--webserver_port=0
--builtin_ntp_servers=127.27.75.212:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:42841
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/testchainca.cert with env {}
W20260923 18:11:08.048455 940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:08.048741 940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:08.048785 940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:08.052980 940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:08.053045 940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:08.053059 940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:08.053078 940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:08.053092 940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 18:11:08.053110 940 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 18:11:08.053158 940 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 18:11:08.058655 940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:42841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:08.059737 940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:08.060729 940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:08.069073 946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:08.069242 940 server_base.cc:1034] running on GCE node
W20260923 18:11:08.069190 948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:08.069078 945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:08.070016 940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:08.071249 940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:08.072477 940 hybrid_clock.cc:648] HybridClock initialized: now 1790187068072450 us; error 31 us; skew 500 ppm
I20260923 18:11:08.074551 940 webserver.cc:460] Webserver started at http://127.27.75.254:46329/ using document root <none> and password file <none>
I20260923 18:11:08.075208 940 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:08.075273 940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:08.075440 940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:08.077615 940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "73f1326db8244616ab9f556a80f0c6fe"
format_stamp: "Formatted at 2026-09-23 18:11:08 on dist-test-slave-wsx5"
server_key: "cca502d4a4d52d56d197cfff123751f3"
server_key_iv: "461c02e94d3764b2622819ebae2fb9d1"
server_key_version: "encryptionkey@0"
I20260923 18:11:08.078167 940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "73f1326db8244616ab9f556a80f0c6fe"
format_stamp: "Formatted at 2026-09-23 18:11:08 on dist-test-slave-wsx5"
server_key: "cca502d4a4d52d56d197cfff123751f3"
server_key_iv: "461c02e94d3764b2622819ebae2fb9d1"
server_key_version: "encryptionkey@0"
I20260923 18:11:08.081959 940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:11:08.084497 954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:08.085911 940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 18:11:08.086036 940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "73f1326db8244616ab9f556a80f0c6fe"
format_stamp: "Formatted at 2026-09-23 18:11:08 on dist-test-slave-wsx5"
server_key: "cca502d4a4d52d56d197cfff123751f3"
server_key_iv: "461c02e94d3764b2622819ebae2fb9d1"
server_key_version: "encryptionkey@0"
I20260923 18:11:08.086148 940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:08.112541 940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:08.113376 940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:08.113577 940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:08.120227 940 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:42841
I20260923 18:11:08.120272 1007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:42841 every 8 connection(s)
I20260923 18:11:08.121268 940 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:08.123348 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 940
I20260923 18:11:08.123449 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:08.123726 27951 external_mini_cluster.cc:1428] Setting key e68f28fe8eff077cfbbde5d5381d7bd9
I20260923 18:11:08.124320 1008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:08.131443 1008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe: Bootstrap starting.
I20260923 18:11:08.134140 1011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:08.124968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.27.75.254:42841)
0923 18:11:08.125689 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:08.125710 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:08.125755 (+ 45us) server_negotiation.cc:407] Connection header received
0923 18:11:08.126524 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:08.126548 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:08.127066 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:08.127444 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:08.127909 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:08.129356 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:08.130301 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:08.130985 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:08.131137 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:08.131980 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:08.131987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:08.132003 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:08.132493 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:08.132565 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:08.132594 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:08.132725 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:08.132838 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 18:11:08.133154 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":168,"threads_started":1}
I20260923 18:11:08.134306 1008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:08.134995 1008 log.cc:824] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:08.137039 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:11:08.137145 1008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe: No bootstrap required, opened a new log
I20260923 18:11:08.140787 1008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f1326db8244616ab9f556a80f0c6fe" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42841 } }
I20260923 18:11:08.141028 1008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:08.141064 1008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f1326db8244616ab9f556a80f0c6fe, State: Initialized, Role: FOLLOWER
I20260923 18:11:08.141507 1008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f1326db8244616ab9f556a80f0c6fe" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42841 } }
I20260923 18:11:08.141633 1008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:08.141681 1008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:08.141753 1008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:08.142652 1008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f1326db8244616ab9f556a80f0c6fe" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42841 } }
I20260923 18:11:08.142952 1008 leader_election.cc:302] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73f1326db8244616ab9f556a80f0c6fe; no voters:
I20260923 18:11:08.143218 1008 leader_election.cc:288] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:08.143334 1013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:08.143539 1013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [term 1 LEADER]: Becoming Leader. State: Replica: 73f1326db8244616ab9f556a80f0c6fe, State: Running, Role: LEADER
I20260923 18:11:08.143838 1013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f1326db8244616ab9f556a80f0c6fe" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42841 } }
I20260923 18:11:08.144348 1008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:08.145486 1014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73f1326db8244616ab9f556a80f0c6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f1326db8244616ab9f556a80f0c6fe" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42841 } } }
I20260923 18:11:08.145592 1014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:08.145911 1015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73f1326db8244616ab9f556a80f0c6fe. Latest consensus state: current_term: 1 leader_uuid: "73f1326db8244616ab9f556a80f0c6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f1326db8244616ab9f556a80f0c6fe" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42841 } } }
I20260923 18:11:08.146013 1015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:08.145974 1022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:08.148664 1022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:08.154636 1022 catalog_manager.cc:1347] Generated new cluster ID: 99d9e36b32aa402c9f896fb2f4374ed7
I20260923 18:11:08.154709 1022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:08.160408 1022 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:08.161419 1022 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:08.165647 1022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73f1326db8244616ab9f556a80f0c6fe: Generated new TSK 0
I20260923 18:11:08.166352 1022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:11:08.340670 1011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:08.177570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.27.75.254:42841)
0923 18:11:08.177703 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:08.177708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:08.177947 (+ 239us) server_negotiation.cc:407] Connection header received
0923 18:11:08.178114 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:08.178117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:08.178166 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:08.178268 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0923 18:11:08.178495 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:08.179034 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:08.179898 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:08.180114 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:08.180199 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:08.180506 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 18:11:08.340174 (+159668us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 18:11:08.340509 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38130: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 18:11:08.340770 1011 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38130: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 18:11:08.340829 1037 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:08.177483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42841 (local address 127.0.0.1:38130)
0923 18:11:08.177864 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:08.177876 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:08.177982 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:08.178276 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:08.178280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:08.178287 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 18:11:08.178396 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:08.178403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:08.179154 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:08.179158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:08.179790 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:08.179798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:08.180225 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:08.180229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:08.180325 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:08.180349 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 18:11:08.340440 (+160091us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 18:11:08.340647 (+ 207us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.75.254:42841: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":98,"threads_started":1}
I20260923 18:11:08.341517 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 940
2026-09-23T18:11:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (430 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (125 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:11:08 dist-test-slave-wsx5 krb5kdc[1049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:11:08 dist-test-slave-wsx5 krb5kdc[1049](info): set up 2 sockets
Sep 23 18:11:08 dist-test-slave-wsx5 krb5kdc[1049](info): commencing operation
krb5kdc: starting...
W20260923 18:11:09.535693 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:11:09 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:11:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:11:09.656992 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:46285
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:46285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:09.774679 1062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:09.775051 1062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:09.775130 1062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:09.779312 1062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:09.779386 1062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:09.779405 1062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:09.779422 1062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:09.779436 1062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:09.785053 1062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:46285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:46285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:09.786191 1062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:09.787262 1062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:11:09.796497 1062 server_base.cc:1034] running on GCE node
W20260923 18:11:09.796432 1067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:09.796418 1070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:09.796419 1068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:09.797387 1062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:09.798589 1062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:09.799770 1062 hybrid_clock.cc:648] HybridClock initialized: now 1790187069799733 us; error 56 us; skew 500 ppm
Sep 23 18:11:09 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187069, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:09.806118 1062 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:11:09.807106 1062 webserver.cc:460] Webserver started at http://127.27.75.254:34491/ using document root <none> and password file <none>
I20260923 18:11:09.807644 1062 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:09.807714 1062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:09.807889 1062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:09.810067 1062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "79faa40bf2454ae99d6a30688501aa04"
format_stamp: "Formatted at 2026-09-23 18:11:09 on dist-test-slave-wsx5"
server_key: "c17bbbbf14852bcda79c653e3456985e"
server_key_iv: "6f50a86eac4910d34e1976b3bf591e33"
server_key_version: "encryptionkey@0"
I20260923 18:11:09.810667 1062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "79faa40bf2454ae99d6a30688501aa04"
format_stamp: "Formatted at 2026-09-23 18:11:09 on dist-test-slave-wsx5"
server_key: "c17bbbbf14852bcda79c653e3456985e"
server_key_iv: "6f50a86eac4910d34e1976b3bf591e33"
server_key_version: "encryptionkey@0"
I20260923 18:11:09.814869 1062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:11:09.817623 1077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:09.819134 1062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:09.819278 1062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "79faa40bf2454ae99d6a30688501aa04"
format_stamp: "Formatted at 2026-09-23 18:11:09 on dist-test-slave-wsx5"
server_key: "c17bbbbf14852bcda79c653e3456985e"
server_key_iv: "6f50a86eac4910d34e1976b3bf591e33"
server_key_version: "encryptionkey@0"
I20260923 18:11:09.819386 1062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:09.843681 1062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:09.847200 1062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:09.847368 1062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:09.854131 1062 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:46285
I20260923 18:11:09.854149 1129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:46285 every 8 connection(s)
I20260923 18:11:09.855104 1062 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:09.857868 1130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:09.863441 1130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04: Bootstrap starting.
I20260923 18:11:09.863787 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 1062
I20260923 18:11:09.863905 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:09.864188 27951 external_mini_cluster.cc:1428] Setting key eb5191953eaf01e78db64f141e7cb274
I20260923 18:11:09.866050 1130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:09.866796 1130 log.cc:824] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:09.868709 1130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04: No bootstrap required, opened a new log
I20260923 18:11:09.872303 1130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79faa40bf2454ae99d6a30688501aa04" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46285 } }
I20260923 18:11:09.872555 1130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:09.872623 1130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79faa40bf2454ae99d6a30688501aa04, State: Initialized, Role: FOLLOWER
Sep 23 18:11:09 dist-test-slave-wsx5 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:09.873070 1130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79faa40bf2454ae99d6a30688501aa04" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46285 } }
I20260923 18:11:09.873301 1130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:09.873396 1130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:09.873503 1130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:09.874579 1130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79faa40bf2454ae99d6a30688501aa04" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46285 } }
I20260923 18:11:09.874959 1130 leader_election.cc:302] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79faa40bf2454ae99d6a30688501aa04; no voters:
I20260923 18:11:09.875303 1130 leader_election.cc:288] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:09.875396 1135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:09.875556 1135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [term 1 LEADER]: Becoming Leader. State: Replica: 79faa40bf2454ae99d6a30688501aa04, State: Running, Role: LEADER
I20260923 18:11:09.875914 1135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79faa40bf2454ae99d6a30688501aa04" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46285 } }
I20260923 18:11:09.876638 1130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:09.877230 1135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79faa40bf2454ae99d6a30688501aa04. Latest consensus state: current_term: 1 leader_uuid: "79faa40bf2454ae99d6a30688501aa04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79faa40bf2454ae99d6a30688501aa04" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46285 } } }
I20260923 18:11:09.877787 1135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:09.878165 1136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79faa40bf2454ae99d6a30688501aa04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79faa40bf2454ae99d6a30688501aa04" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 46285 } } }
I20260923 18:11:09.878955 1136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:09.879415 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:09.865645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.27.75.254:46285)
0923 18:11:09.866071 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:09.866087 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:09.866121 (+ 34us) server_negotiation.cc:407] Connection header received
0923 18:11:09.866829 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:09.866847 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:09.867219 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:09.867568 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:09.868366 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:09.869458 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:09.870519 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:09.871040 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:09.871195 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:09.873833 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:09.873838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:09.873853 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:09.873886 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:09.875642 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:09.876148 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:09.876153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:09.876158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:09.876277 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:09.876540 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:09.876544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:09.876546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:09.877005 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:09.877819 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:09.877975 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 18:11:09.878297 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":112,"threads_started":1}
W20260923 18:11:09.882761 1150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:11:09.882858 1150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:11:09.882968 1146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:09.883816 1146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:09.889854 1146 catalog_manager.cc:1347] Generated new cluster ID: 41ed38cefa844ff887311c63bca21da1
I20260923 18:11:09.889930 1146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:09.919739 1146 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:09.920740 1146 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:09.927806 1146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79faa40bf2454ae99d6a30688501aa04: Generated new TSK 0
I20260923 18:11:09.928503 1146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:11:09.970053 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:46285
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:10.095181 1156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:10.095480 1156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:10.095523 1156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:10.100004 1156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:10.100090 1156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:10.100169 1156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:10.105698 1156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:10.106815 1156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:10.107838 1156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:10.117931 1161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:10.118089 1156 server_base.cc:1034] running on GCE node
W20260923 18:11:10.117942 1162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:10.117969 1164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:10.118830 1156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:10.119529 1156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:10.120734 1156 hybrid_clock.cc:648] HybridClock initialized: now 1790187070120700 us; error 42 us; skew 500 ppm
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:10.124004 1156 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:11:10.125001 1156 webserver.cc:460] Webserver started at http://127.27.75.193:43421/ using document root <none> and password file <none>
I20260923 18:11:10.125617 1156 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:10.125687 1156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:10.125892 1156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:10.127921 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "08de352601994f0ab5a178a3cab7cd91"
format_stamp: "Formatted at 2026-09-23 18:11:10 on dist-test-slave-wsx5"
server_key: "3ef76da448ce5b0bbf69b832cf1e13f5"
server_key_iv: "499c2e04fc2a7539d04cae9eb974419a"
server_key_version: "encryptionkey@0"
I20260923 18:11:10.128472 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "08de352601994f0ab5a178a3cab7cd91"
format_stamp: "Formatted at 2026-09-23 18:11:10 on dist-test-slave-wsx5"
server_key: "3ef76da448ce5b0bbf69b832cf1e13f5"
server_key_iv: "499c2e04fc2a7539d04cae9eb974419a"
server_key_version: "encryptionkey@0"
I20260923 18:11:10.132597 1156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:11:10.135470 1171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:10.137171 1156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:11:10.137334 1156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "08de352601994f0ab5a178a3cab7cd91"
format_stamp: "Formatted at 2026-09-23 18:11:10 on dist-test-slave-wsx5"
server_key: "3ef76da448ce5b0bbf69b832cf1e13f5"
server_key_iv: "499c2e04fc2a7539d04cae9eb974419a"
server_key_version: "encryptionkey@0"
I20260923 18:11:10.137468 1156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:10.171082 1156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:10.175078 1156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:10.175302 1156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:10.175943 1156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:10.176829 1156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:10.176878 1156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:10.176934 1156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:10.176961 1156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:10.185807 1156 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:43551
I20260923 18:11:10.185827 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:43551 every 8 connection(s)
I20260923 18:11:10.186772 1156 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:10.187961 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 1156
I20260923 18:11:10.188087 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:11:10.188410 27951 external_mini_cluster.cc:1428] Setting key 14dd478e62e4712195439218e53439df
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:10.207176 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:10.188848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:41595 (local address 127.27.75.254:46285)
0923 18:11:10.188988 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:10.188993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:10.190112 (+ 1119us) server_negotiation.cc:407] Connection header received
0923 18:11:10.191644 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:10.191648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:10.191702 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:10.191780 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:10.193669 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.194288 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.196208 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.196473 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.196576 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:10.202380 (+ 5804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:10.202384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:10.202386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:10.202418 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:10.204410 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.205222 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.205226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.205228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.205339 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.205782 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.205786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.205787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.206050 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:10.206170 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:10.206805 (+ 635us) server_negotiation.cc:299] Negotiation successful
0923 18:11:10.206976 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:11:10.207990 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:10.189353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:46285 (local address 127.27.75.193:41595)
0923 18:11:10.189935 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:10.190005 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:10.191359 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:10.192044 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:10.192055 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:10.192643 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:10.193446 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:10.193469 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.194454 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.194458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:10.196018 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:10.196034 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.196668 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.196673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:10.196915 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:10.197652 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:10.197678 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:10.202099 (+ 4421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:10.204643 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:10.204649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:10.204668 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:10.205015 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:10.205484 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:10.205487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:10.205489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:10.205670 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:10.206187 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:10.206193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:10.206497 (+ 304us) client_negotiation.cc:769] Sending connection context
0923 18:11:10.206688 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 18:11:10.206998 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":143,"threads_started":1}
I20260923 18:11:10.209362 1285 heartbeater.cc:347] Connected to a master server at 127.27.75.254:46285
I20260923 18:11:10.209609 1285 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:10.210254 1285 heartbeater.cc:511] Master 127.27.75.254:46285 requested a full tablet report, sending...
I20260923 18:11:10.211957 1094 ts_manager.cc:194] Registered new tserver with Master: 08de352601994f0ab5a178a3cab7cd91 (127.27.75.193:43551)
I20260923 18:11:10.213384 1094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:41595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:11:10.221809 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:46285
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:10.341013 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:10.341398 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:10.341445 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:10.345919 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:10.346005 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:10.346067 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:10.351841 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:10.352955 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:10.354063 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:10.363859 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:10.363926 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:10.363994 1290 server_base.cc:1034] running on GCE node
W20260923 18:11:10.363917 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:10.364579 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:10.365243 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:10.366432 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790187070366401 us; error 41 us; skew 500 ppm
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:10.369457 1290 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:11:10.370381 1290 webserver.cc:460] Webserver started at http://127.27.75.194:43885/ using document root <none> and password file <none>
I20260923 18:11:10.370931 1290 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:10.370991 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:10.371217 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:10.373335 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "d10ea402994d464fbeda15f7008afa83"
format_stamp: "Formatted at 2026-09-23 18:11:10 on dist-test-slave-wsx5"
server_key: "fb2e3ca67a2279508d2a1609d6e5732c"
server_key_iv: "876cdc60a8cd1d813cd1f8acc98e3d07"
server_key_version: "encryptionkey@0"
I20260923 18:11:10.373950 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "d10ea402994d464fbeda15f7008afa83"
format_stamp: "Formatted at 2026-09-23 18:11:10 on dist-test-slave-wsx5"
server_key: "fb2e3ca67a2279508d2a1609d6e5732c"
server_key_iv: "876cdc60a8cd1d813cd1f8acc98e3d07"
server_key_version: "encryptionkey@0"
I20260923 18:11:10.378120 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:11:10.380803 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:10.382297 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:10.382436 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "d10ea402994d464fbeda15f7008afa83"
format_stamp: "Formatted at 2026-09-23 18:11:10 on dist-test-slave-wsx5"
server_key: "fb2e3ca67a2279508d2a1609d6e5732c"
server_key_iv: "876cdc60a8cd1d813cd1f8acc98e3d07"
server_key_version: "encryptionkey@0"
I20260923 18:11:10.382556 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:10.398033 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:10.401392 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:10.401593 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:10.402189 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:10.403050 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:10.403101 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:10.403155 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:10.403182 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:10.411976 1290 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:33023
I20260923 18:11:10.412006 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:33023 every 8 connection(s)
I20260923 18:11:10.412921 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:10.418416 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 1290
I20260923 18:11:10.418529 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:11:10.418813 27951 external_mini_cluster.cc:1428] Setting key d104168c5008537aa7003c23fccf5906
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:10.428254 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:10.415003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:51737 (local address 127.27.75.254:46285)
0923 18:11:10.415145 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:10.415149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:10.416041 (+ 892us) server_negotiation.cc:407] Connection header received
0923 18:11:10.417481 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:10.417484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:10.417543 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:10.417629 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:10.419134 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.419805 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.421429 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.421670 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.421768 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:10.424528 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:10.424532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:10.424534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:10.424558 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:10.425946 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.426549 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.426553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.426554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.426636 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.427045 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.427047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.427049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.427238 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:10.427314 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:10.428009 (+ 695us) server_negotiation.cc:299] Negotiation successful
0923 18:11:10.428120 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:11:10.429245 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:10.415421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:46285 (local address 127.27.75.194:51737)
0923 18:11:10.415888 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:10.415933 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:10.417263 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:10.417807 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:10.417814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:10.418248 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:10.418963 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:10.418985 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.419955 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.419958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:10.421278 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:10.421291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.421826 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.421829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:10.422028 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:10.422713 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:10.422764 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:10.424346 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:10.426059 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:10.426065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:10.426084 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:10.426442 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:10.426767 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:10.426770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:10.426772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:10.426962 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:10.427340 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:10.427347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:10.427682 (+ 335us) client_negotiation.cc:769] Sending connection context
0923 18:11:10.427959 (+ 277us) client_negotiation.cc:240] Negotiation successful
0923 18:11:10.428321 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260923 18:11:10.430601 1419 heartbeater.cc:347] Connected to a master server at 127.27.75.254:46285
I20260923 18:11:10.430902 1419 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
I20260923 18:11:10.431547 1419 heartbeater.cc:511] Master 127.27.75.254:46285 requested a full tablet report, sending...
I20260923 18:11:10.432785 1094 ts_manager.cc:194] Registered new tserver with Master: d10ea402994d464fbeda15f7008afa83 (127.27.75.194:33023)
I20260923 18:11:10.433585 1094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:51737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:11:10.449268 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:46285
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:10.565411 1424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:10.565726 1424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:10.565773 1424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:10.570288 1424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:10.570376 1424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:10.570439 1424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:10.576403 1424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:46285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:10.577713 1424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:10.578867 1424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:10.589792 1429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:10.589859 1432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:10.589859 1430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:10.590137 1424 server_base.cc:1034] running on GCE node
I20260923 18:11:10.590601 1424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:10.591308 1424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:10.592537 1424 hybrid_clock.cc:648] HybridClock initialized: now 1790187070592462 us; error 81 us; skew 500 ppm
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:10.596038 1424 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:11:10.597020 1424 webserver.cc:460] Webserver started at http://127.27.75.195:46335/ using document root <none> and password file <none>
I20260923 18:11:10.597680 1424 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:10.597759 1424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:10.597978 1424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:10.600260 1424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "6f05e3db4aa3475090a0a39f5cd27034"
format_stamp: "Formatted at 2026-09-23 18:11:10 on dist-test-slave-wsx5"
server_key: "98c88d66c85509d012e43aca4e235e4e"
server_key_iv: "d31faa884a6d47b8d040667da74681b7"
server_key_version: "encryptionkey@0"
I20260923 18:11:10.600868 1424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "6f05e3db4aa3475090a0a39f5cd27034"
format_stamp: "Formatted at 2026-09-23 18:11:10 on dist-test-slave-wsx5"
server_key: "98c88d66c85509d012e43aca4e235e4e"
server_key_iv: "d31faa884a6d47b8d040667da74681b7"
server_key_version: "encryptionkey@0"
I20260923 18:11:10.605036 1424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 18:11:10.607810 1439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:10.609421 1424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 18:11:10.609575 1424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "6f05e3db4aa3475090a0a39f5cd27034"
format_stamp: "Formatted at 2026-09-23 18:11:10 on dist-test-slave-wsx5"
server_key: "98c88d66c85509d012e43aca4e235e4e"
server_key_iv: "d31faa884a6d47b8d040667da74681b7"
server_key_version: "encryptionkey@0"
I20260923 18:11:10.609697 1424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:10.627074 1424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:10.630272 1424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:10.630460 1424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:10.631067 1424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:10.631884 1424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:10.631930 1424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:10.631982 1424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:10.632011 1424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:10.640527 1424 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:35201
I20260923 18:11:10.640542 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:35201 every 8 connection(s)
I20260923 18:11:10.641597 1424 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:10.646301 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 1424
I20260923 18:11:10.646430 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:11:10.646904 27951 external_mini_cluster.cc:1428] Setting key b2e2a74ce27f23fa38ce10e064097464
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:10.656733 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:10.643589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:45685 (local address 127.27.75.254:46285)
0923 18:11:10.643745 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:10.643748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:10.644709 (+ 961us) server_negotiation.cc:407] Connection header received
0923 18:11:10.646216 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:10.646220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:10.646286 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:10.646360 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:10.647839 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.648554 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.650197 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.650427 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.650529 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:10.653031 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:10.653035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:10.653037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:10.653064 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:10.654507 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.655113 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.655116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.655117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.655208 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.655637 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.655640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.655642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.655855 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:10.655934 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:10.656490 (+ 556us) server_negotiation.cc:299] Negotiation successful
0923 18:11:10.656597 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 18:11:10.657713 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:10.643964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:46285 (local address 127.27.75.195:45685)
0923 18:11:10.644502 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:10.644550 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:10.645934 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:10.646586 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:10.646595 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:10.647005 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:10.647636 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:10.647664 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.648766 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.648770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:10.650069 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:10.650081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.650572 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.650575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:10.650768 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:10.651328 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:10.651345 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:10.652849 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:10.654636 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:10.654641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:10.654695 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:10.654977 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:10.655338 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:10.655342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:10.655343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:10.655521 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:10.655968 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:10.655979 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:10.656274 (+ 295us) client_negotiation.cc:769] Sending connection context
0923 18:11:10.656447 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 18:11:10.656740 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":131,"threads_started":1}
I20260923 18:11:10.659011 1553 heartbeater.cc:347] Connected to a master server at 127.27.75.254:46285
I20260923 18:11:10.659260 1553 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:10.659765 1553 heartbeater.cc:511] Master 127.27.75.254:46285 requested a full tablet report, sending...
I20260923 18:11:10.660859 1094 ts_manager.cc:194] Registered new tserver with Master: 6f05e3db4aa3475090a0a39f5cd27034 (127.27.75.195:35201)
I20260923 18:11:10.661532 1094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:45685
I20260923 18:11:10.661656 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/test-admin.keytab.
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:10.679453 27951 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 18:11:10.682135 1560 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:10.680848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.27.75.193:43551)
0923 18:11:10.681154 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:10.681173 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:10.681200 (+ 27us) server_negotiation.cc:407] Connection header received
0923 18:11:10.681250 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:10.681255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:10.681447 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:10.681565 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:10.681953 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55180: BlockingRecv error: recv got EOF from 127.0.0.1:55180 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
W20260923 18:11:10.682435 27951 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5cc)
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:10.683419 27951 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:10.684381 27951 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:11:10.694236 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:10.685290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.27.75.193:43551)
0923 18:11:10.685426 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:10.685431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:10.685495 (+ 64us) server_negotiation.cc:407] Connection header received
0923 18:11:10.685614 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:10.685616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:10.685657 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:10.685723 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:10.686212 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.687008 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.687879 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.688278 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.688360 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:10.690847 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:10.690857 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:10.690871 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:10.690899 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:10.692484 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.692900 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.692904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.692906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.692983 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.693249 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.693251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.693253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.693506 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:10.693672 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:10.693906 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 18:11:10.694049 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 18:11:10.698045 1560 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:10.697009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.27.75.193:43551)
0923 18:11:10.697165 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:10.697170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:10.697284 (+ 114us) server_negotiation.cc:407] Connection header received
0923 18:11:10.697386 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:10.697389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:10.697433 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:10.697495 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:10.697930 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55206: BlockingRecv error: recv got EOF from 127.0.0.1:55206 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260923 18:11:10.698127 27951 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5cc' of type FILE
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:10.699142 27951 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:10.700059 27951 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:11:10.709086 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:10.700947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.27.75.193:43551)
0923 18:11:10.701156 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:10.701160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:10.701177 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:11:10.701332 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:10.701335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:10.701376 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:10.701443 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:10.701916 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.702463 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.703218 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.703467 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.703552 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:10.706002 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:10.706005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:10.706007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:10.706034 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:10.707311 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.707799 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.707802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.707803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.707889 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.708174 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.708177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.708178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.708382 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:10.708491 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:10.708797 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 18:11:10.708922 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260923 18:11:10.712010 1560 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:10.710923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.27.75.193:43551)
0923 18:11:10.711046 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:10.711050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:10.711142 (+ 92us) server_negotiation.cc:407] Connection header received
0923 18:11:10.711269 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:10.711272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:10.711313 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:10.711379 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:10.711903 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55228: BlockingRecv error: recv got EOF from 127.0.0.1:55228 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 18:11:10.712141 27951 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790187042487232-27951-0/krb5kdc/krb5cc' of type FILE
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:10.713318 27951 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:10.714288 27951 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:11:10.723316 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:10.715274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.27.75.193:43551)
0923 18:11:10.715404 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:10.715409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:10.715492 (+ 83us) server_negotiation.cc:407] Connection header received
0923 18:11:10.715629 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:10.715632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:10.715678 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:10.715740 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:10.716279 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.716772 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.717631 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:10.717858 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:10.717948 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:10.720287 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:10.720291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:10.720294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:10.720322 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:10.721744 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.722145 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.722149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.722151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.722238 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:10.722526 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:10.722528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:10.722529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:10.722693 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:10.722787 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:10.723087 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 18:11:10.723193 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:11:10.724442 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 1156
I20260923 18:11:10.729568 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 1290
I20260923 18:11:10.735031 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 1424
I20260923 18:11:10.739957 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 1062
2026-09-23T18:11:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2154 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1573](info): set up 2 sockets
Sep 23 18:11:10 dist-test-slave-wsx5 krb5kdc[1573](info): commencing operation
krb5kdc: starting...
W20260923 18:11:11.699958 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:11:11 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:11:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.254.keytab.
Entry for principal oryx/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.254.keytab.
I20260923 18:11:11.828296 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:43469
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.254.keytab
--principal=oryx/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:43469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:11.940371 1586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:11.940652 1586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:11.940694 1586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:11.944772 1586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:11.944836 1586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:11.944854 1586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:11.944869 1586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:11.944883 1586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:11.950282 1586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:43469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.254.keytab
--principal=oryx/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:11.951324 1586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:11.952289 1586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:11.960927 1592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:11.960920 1591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:11.960920 1594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:11.961234 1586 server_base.cc:1034] running on GCE node
I20260923 18:11:11.961809 1586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:11.962993 1586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:11.964174 1586 hybrid_clock.cc:648] HybridClock initialized: now 1790187071964145 us; error 44 us; skew 500 ppm
Sep 23 18:11:11 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187071, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:11.967214 1586 init.cc:377] Logged in from keytab as oryx/127.27.75.254@KRBTEST.COM (short username oryx)
I20260923 18:11:11.968235 1586 webserver.cc:460] Webserver started at http://127.27.75.254:40035/ using document root <none> and password file <none>
I20260923 18:11:11.968847 1586 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:11.968918 1586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:11.969156 1586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:11.971249 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "e3a6b8ab2c1340c8a837f723566263c0"
format_stamp: "Formatted at 2026-09-23 18:11:11 on dist-test-slave-wsx5"
server_key: "583e1dbe2ffbf4f6cf09bfe2a4ee8244"
server_key_iv: "2a582a2861788105ed8a65503ef33bc5"
server_key_version: "encryptionkey@0"
I20260923 18:11:11.971812 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "e3a6b8ab2c1340c8a837f723566263c0"
format_stamp: "Formatted at 2026-09-23 18:11:11 on dist-test-slave-wsx5"
server_key: "583e1dbe2ffbf4f6cf09bfe2a4ee8244"
server_key_iv: "2a582a2861788105ed8a65503ef33bc5"
server_key_version: "encryptionkey@0"
I20260923 18:11:11.975745 1586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 18:11:11.978284 1601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:11.979614 1586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:11.979744 1586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "e3a6b8ab2c1340c8a837f723566263c0"
format_stamp: "Formatted at 2026-09-23 18:11:11 on dist-test-slave-wsx5"
server_key: "583e1dbe2ffbf4f6cf09bfe2a4ee8244"
server_key_iv: "2a582a2861788105ed8a65503ef33bc5"
server_key_version: "encryptionkey@0"
I20260923 18:11:11.979851 1586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:12.002519 1586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:12.005733 1586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:12.005923 1586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:12.012897 1586 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:43469
I20260923 18:11:12.013192 1653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:43469 every 8 connection(s)
I20260923 18:11:12.014110 1586 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:12.014981 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 1586
I20260923 18:11:12.015118 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:12.015434 27951 external_mini_cluster.cc:1428] Setting key 7214379405d1dedce52395c88ec4a86e
I20260923 18:11:12.017556 1654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:12.023969 1654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0: Bootstrap starting.
Sep 23 18:11:12 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
I20260923 18:11:12.026374 1654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:12.027068 1654 log.cc:824] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:12.029012 1654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0: No bootstrap required, opened a new log
I20260923 18:11:12.029847 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.016908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.27.75.254:43469)
0923 18:11:12.017346 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:12.017362 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:12.017399 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:11:12.018099 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:12.018118 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:12.018457 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:12.018748 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:12.019577 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.020597 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.021741 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.022258 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.022391 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.025187 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:12.025193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:12.025206 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:12.025234 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:12.026936 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.027417 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.027423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.027428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.027542 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.027823 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.027826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.027827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.028251 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:12.028427 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:12.028703 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 18:11:12.028991 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":139,"threads_started":1}
I20260923 18:11:12.032905 1654 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } }
I20260923 18:11:12.033178 1654 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:12.033241 1654 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a6b8ab2c1340c8a837f723566263c0, State: Initialized, Role: FOLLOWER
I20260923 18:11:12.033723 1654 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } }
I20260923 18:11:12.033880 1654 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:12.033948 1654 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:12.034044 1654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:12.035005 1654 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } }
I20260923 18:11:12.035337 1654 leader_election.cc:302] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3a6b8ab2c1340c8a837f723566263c0; no voters:
I20260923 18:11:12.035637 1654 leader_election.cc:288] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:12.035791 1659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:12.036010 1659 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 1 LEADER]: Becoming Leader. State: Replica: e3a6b8ab2c1340c8a837f723566263c0, State: Running, Role: LEADER
I20260923 18:11:12.036321 1659 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } }
I20260923 18:11:12.036876 1654 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:12.038266 1661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3a6b8ab2c1340c8a837f723566263c0. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } }
I20260923 18:11:12.038391 1661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:12.038350 1660 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } }
I20260923 18:11:12.038443 1660 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:12.039568 1668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:12.042130 1668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:12.048146 1668 catalog_manager.cc:1347] Generated new cluster ID: af16d14630074fcb8a236c9641704d69
I20260923 18:11:12.048218 1668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:12.056210 1668 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:12.057502 1668 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:12.062441 1668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0: Generated new TSK 0
I20260923 18:11:12.063282 1668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab.
Entry for principal oryx/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab.
I20260923 18:11:12.096889 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43469
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab
--principal=oryx/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:12.208177 1680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:12.208451 1680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:12.208495 1680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:12.212580 1680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:12.212661 1680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:12.212723 1680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:12.218132 1680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab
--principal=oryx/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:12.219234 1680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:12.220218 1680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:12.229840 1688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:12.229890 1680 server_base.cc:1034] running on GCE node
W20260923 18:11:12.229840 1686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:12.230108 1685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:12.230508 1680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:12.231182 1680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:12.232419 1680 hybrid_clock.cc:648] HybridClock initialized: now 1790187072232395 us; error 32 us; skew 500 ppm
Sep 23 18:11:12 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187072, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:12.235549 1680 init.cc:377] Logged in from keytab as oryx/127.27.75.193@KRBTEST.COM (short username oryx)
I20260923 18:11:12.236466 1680 webserver.cc:460] Webserver started at http://127.27.75.193:33945/ using document root <none> and password file <none>
I20260923 18:11:12.237028 1680 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:12.237092 1680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:12.237423 1680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:12.239555 1680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "5f45b3b090e44171bed8b23947a2bb7a"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "bc74d4820a20b1db431ffe3b0c1db47f"
server_key_iv: "878f6f7ec9dd64751612b62a44c52d92"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.240134 1680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "5f45b3b090e44171bed8b23947a2bb7a"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "bc74d4820a20b1db431ffe3b0c1db47f"
server_key_iv: "878f6f7ec9dd64751612b62a44c52d92"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.244122 1680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:11:12.248462 1695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:12.249943 1680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:11:12.250068 1680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "5f45b3b090e44171bed8b23947a2bb7a"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "bc74d4820a20b1db431ffe3b0c1db47f"
server_key_iv: "878f6f7ec9dd64751612b62a44c52d92"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.250216 1680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:12.265280 1680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:12.268833 1680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:12.269030 1680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:12.269665 1680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:12.270474 1680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:12.270520 1680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:12.270558 1680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:12.270583 1680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:12.278780 1680 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:44753
I20260923 18:11:12.278810 1808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:44753 every 8 connection(s)
I20260923 18:11:12.279687 1680 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:12.283296 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 1680
I20260923 18:11:12.283419 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:11:12.283784 27951 external_mini_cluster.cc:1428] Setting key 965efea8200a9bf16935d41126379e55
Sep 23 18:11:12 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187072, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:11:12.294624 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.281493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:50537 (local address 127.27.75.254:43469)
0923 18:11:12.281639 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:12.281642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:12.282435 (+ 793us) server_negotiation.cc:407] Connection header received
0923 18:11:12.284034 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:12.284038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:12.284099 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:12.284171 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:12.285656 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.286216 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.287850 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.288097 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.288185 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.290729 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:12.290733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:12.290735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:12.290761 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:12.292350 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.292934 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.292938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.292940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.293019 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.293378 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.293382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.293384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.293631 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:12.293727 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:12.294254 (+ 527us) server_negotiation.cc:299] Negotiation successful
0923 18:11:12.294420 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.194@KRBTEST.COM" created.
I20260923 18:11:12.295701 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.281843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.193:50537)
0923 18:11:12.282285 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:12.282329 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:12.283810 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:12.284373 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:12.284381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:12.284843 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:12.285470 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:12.285486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.286349 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.286352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:12.287726 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:12.287738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.288242 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.288246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:12.288424 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.289061 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:12.289079 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:12.290553 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:12.292476 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:12.292482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:12.292499 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:12.292840 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:12.293100 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:12.293103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:12.293105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:12.293289 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:12.293724 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:12.293731 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:12.293996 (+ 265us) client_negotiation.cc:769] Sending connection context
0923 18:11:12.294172 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 18:11:12.294519 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260923 18:11:12.297207 1809 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43469
I20260923 18:11:12.297452 1809 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:12.297955 1809 heartbeater.cc:511] Master 127.27.75.254:43469 requested a full tablet report, sending...
I20260923 18:11:12.299752 1612 ts_manager.cc:194] Registered new tserver with Master: 5f45b3b090e44171bed8b23947a2bb7a (127.27.75.193:44753)
I20260923 18:11:12.301467 1612 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:50537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab.
Entry for principal oryx/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab.
I20260923 18:11:12.313915 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43469
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab
--principal=oryx/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:12.425192 1814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:12.425505 1814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:12.425557 1814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:12.429687 1814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:12.429750 1814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:12.429811 1814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:12.435281 1814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab
--principal=oryx/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:12.436336 1814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:12.437445 1814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:12.447297 1820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:12.447285 1819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:12.447306 1822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:12.447604 1814 server_base.cc:1034] running on GCE node
I20260923 18:11:12.448028 1814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:12.448639 1814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:12.449810 1814 hybrid_clock.cc:648] HybridClock initialized: now 1790187072449787 us; error 33 us; skew 500 ppm
Sep 23 18:11:12 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187072, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:12.452478 1814 init.cc:377] Logged in from keytab as oryx/127.27.75.194@KRBTEST.COM (short username oryx)
I20260923 18:11:12.453434 1814 webserver.cc:460] Webserver started at http://127.27.75.194:32793/ using document root <none> and password file <none>
I20260923 18:11:12.454023 1814 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:12.454083 1814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:12.454289 1814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:12.456386 1814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "6cb3fc9dd9234ff3b6e27343823ef836"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "5f7f473b8a67f7d74a01119e2f0f7cfe"
server_key_iv: "6f687fddbcb4d1f01caa4dce26fea5a2"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.456923 1814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "6cb3fc9dd9234ff3b6e27343823ef836"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "5f7f473b8a67f7d74a01119e2f0f7cfe"
server_key_iv: "6f687fddbcb4d1f01caa4dce26fea5a2"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.460732 1814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:11:12.463214 1829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:12.464476 1814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 18:11:12.464576 1814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "6cb3fc9dd9234ff3b6e27343823ef836"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "5f7f473b8a67f7d74a01119e2f0f7cfe"
server_key_iv: "6f687fddbcb4d1f01caa4dce26fea5a2"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.464687 1814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:12.488353 1814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:12.491343 1814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:12.491514 1814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:12.492084 1814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:12.492908 1814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:12.492950 1814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:12.492993 1814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:12.493021 1814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:12.501223 1814 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:45679
I20260923 18:11:12.501305 1942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:45679 every 8 connection(s)
I20260923 18:11:12.502123 1814 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:12.510429 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 1814
I20260923 18:11:12.510550 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:11:12.510854 27951 external_mini_cluster.cc:1428] Setting key 75556d11a04dddfd602b3bb4052556d4
Sep 23 18:11:12 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187072, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
I20260923 18:11:12.516314 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.503976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:57119 (local address 127.27.75.254:43469)
0923 18:11:12.504131 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:12.504134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:12.504955 (+ 821us) server_negotiation.cc:407] Connection header received
0923 18:11:12.506426 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:12.506429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:12.506480 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:12.506557 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:12.507773 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.508321 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.509568 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.509801 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.509889 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.512595 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:12.512598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:12.512600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:12.512631 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:12.514012 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.514618 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.514662 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.514663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.514750 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.515102 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.515106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.515108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.515363 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:12.515455 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:12.515982 (+ 527us) server_negotiation.cc:299] Negotiation successful
0923 18:11:12.516132 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 18:11:12.517266 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.504342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.194:57119)
0923 18:11:12.504797 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:12.504841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:12.506180 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:12.506737 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:12.506744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:12.507110 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:12.507615 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:12.507629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.508458 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.508461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:12.509443 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:12.509454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.509940 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.509943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:12.510102 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.510767 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:12.510784 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:12.512430 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:12.514126 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:12.514133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:12.514153 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:12.514511 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:12.514834 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:12.514837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:12.514839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:12.514991 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:12.515456 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:12.515461 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:12.515750 (+ 289us) client_negotiation.cc:769] Sending connection context
0923 18:11:12.515907 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 18:11:12.516221 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260923 18:11:12.518509 1943 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43469
I20260923 18:11:12.518761 1943 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:12.519279 1943 heartbeater.cc:511] Master 127.27.75.254:43469 requested a full tablet report, sending...
I20260923 18:11:12.520368 1612 ts_manager.cc:194] Registered new tserver with Master: 6cb3fc9dd9234ff3b6e27343823ef836 (127.27.75.194:45679)
WARNING: no policy specified for oryx/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:11:12.520942 1612 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:57119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab.
Entry for principal oryx/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab.
I20260923 18:11:12.540083 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43469
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab
--principal=oryx/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:12.651404 1948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:12.651681 1948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:12.651746 1948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:12.655907 1948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:12.655975 1948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:12.656059 1948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:12.661497 1948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab
--principal=oryx/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:12.662590 1948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:12.663621 1948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:12.673548 1953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:12.673556 1956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:12.673556 1954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:12.674060 1948 server_base.cc:1034] running on GCE node
I20260923 18:11:12.674460 1948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:12.675124 1948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:12.676306 1948 hybrid_clock.cc:648] HybridClock initialized: now 1790187072676286 us; error 34 us; skew 500 ppm
Sep 23 18:11:12 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187072, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:12.679133 1948 init.cc:377] Logged in from keytab as oryx/127.27.75.195@KRBTEST.COM (short username oryx)
I20260923 18:11:12.680158 1948 webserver.cc:460] Webserver started at http://127.27.75.195:34313/ using document root <none> and password file <none>
I20260923 18:11:12.680732 1948 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:12.680800 1948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:12.680996 1948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:12.683178 1948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "90362291c0604c48ae4d307f55f84b2f"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "4cf226347270dd66ce66a06391031d4a"
server_key_iv: "bff0d3b8e2ac8e8b6f055912bd530369"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.683795 1948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "90362291c0604c48ae4d307f55f84b2f"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "4cf226347270dd66ce66a06391031d4a"
server_key_iv: "bff0d3b8e2ac8e8b6f055912bd530369"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.687777 1948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 18:11:12.690321 1963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:12.691694 1948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:11:12.691805 1948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "90362291c0604c48ae4d307f55f84b2f"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "4cf226347270dd66ce66a06391031d4a"
server_key_iv: "bff0d3b8e2ac8e8b6f055912bd530369"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.691926 1948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:12.710939 1948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:12.714262 1948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:12.714423 1948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:12.714972 1948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:12.715808 1948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:12.715855 1948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:12.715895 1948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:12.715916 1948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:12.724638 1948 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:43129
I20260923 18:11:12.724654 2076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:43129 every 8 connection(s)
I20260923 18:11:12.725653 1948 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:12.726406 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 1948
I20260923 18:11:12.726521 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:11:12.726794 27951 external_mini_cluster.cc:1428] Setting key 66d80c1e585af74ce44c8a49bb293760
Sep 23 18:11:12 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187072, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
I20260923 18:11:12.740302 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.728945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:46769 (local address 127.27.75.254:43469)
0923 18:11:12.729075 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:12.729079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:12.729092 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:12.730343 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:12.730346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:12.730400 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:12.730474 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:12.731749 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.732290 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.733554 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.733789 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.733875 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.736397 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:12.736402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:12.736405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:12.736462 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:12.737967 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.738515 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.738519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.738521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.738628 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.739042 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.739045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.739047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.739296 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:12.739389 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:12.739998 (+ 609us) server_negotiation.cc:299] Negotiation successful
0923 18:11:12.740142 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:11:12.741392 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.727976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.195:46769)
0923 18:11:12.728505 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:12.728564 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:12.730076 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:12.730647 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:12.730657 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:12.731050 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:12.731583 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:12.731599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.732408 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.732410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:12.733409 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:12.733420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.733927 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.733930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:12.734090 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.734680 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:12.734697 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:12.736178 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:12.738100 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:12.738105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:12.738120 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:12.738393 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:12.738755 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:12.738758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:12.738759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:12.738936 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:12.739426 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:12.739434 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:12.739719 (+ 285us) client_negotiation.cc:769] Sending connection context
0923 18:11:12.739928 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 18:11:12.740280 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":119,"threads_started":1}
I20260923 18:11:12.742486 2077 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43469
I20260923 18:11:12.742744 2077 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:12.743273 2077 heartbeater.cc:511] Master 127.27.75.254:43469 requested a full tablet report, sending...
I20260923 18:11:12.744366 1612 ts_manager.cc:194] Registered new tserver with Master: 90362291c0604c48ae4d307f55f84b2f (127.27.75.195:43129)
I20260923 18:11:12.744951 1612 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:46769
I20260923 18:11:12.752887 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.75.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.253.keytab.
Entry for principal oryx/127.27.75.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.253.keytab.
I20260923 18:11:12.781718 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.253:33539
--webserver_interface=127.27.75.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.253.keytab
--principal=oryx/127.27.75.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:43469,127.27.75.253:33539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:12.894016 2082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:12.894316 2082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:12.894376 2082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:12.898581 2082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:12.898655 2082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:12.898684 2082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:12.898705 2082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:12.898732 2082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:12.904182 2082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:43469,127.27.75.253:33539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.253.keytab
--principal=oryx/127.27.75.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.253:33539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.75.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
W20260923 18:11:12.904444 2082 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.75.254:43469,127.27.75.253:33539'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 18:11:12.905385 2082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:12.906407 2082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:12.915060 2090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:12.915078 2087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:12.915078 2088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:12.915185 2082 server_base.cc:1034] running on GCE node
I20260923 18:11:12.915935 2082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:12.917008 2082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:12.918179 2082 hybrid_clock.cc:648] HybridClock initialized: now 1790187072918156 us; error 41 us; skew 500 ppm
Sep 23 18:11:12 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187072, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:12.921205 2082 init.cc:377] Logged in from keytab as oryx/127.27.75.253@KRBTEST.COM (short username oryx)
I20260923 18:11:12.922219 2082 webserver.cc:460] Webserver started at http://127.27.75.253:37457/ using document root <none> and password file <none>
I20260923 18:11:12.922782 2082 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:12.922854 2082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:12.923056 2082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:12.925218 2082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data/instance:
uuid: "761778be5f0f4620ad861bb0b32a7762"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "b4d6199eb82fae9729f432e514f7d54d"
server_key_iv: "827200d1a789f76d85da55cb77e4a63d"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.925794 2082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/wal/instance:
uuid: "761778be5f0f4620ad861bb0b32a7762"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "b4d6199eb82fae9729f432e514f7d54d"
server_key_iv: "827200d1a789f76d85da55cb77e4a63d"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.929735 2082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 18:11:12.932520 2097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:12.934122 2082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:12.934270 2082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/wal
uuid: "761778be5f0f4620ad861bb0b32a7762"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "b4d6199eb82fae9729f432e514f7d54d"
server_key_iv: "827200d1a789f76d85da55cb77e4a63d"
server_key_version: "encryptionkey@0"
I20260923 18:11:12.934391 2082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:12.948647 2082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:12.951911 2082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:12.952106 2082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:12.959237 2082 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.253:33539
I20260923 18:11:12.959633 2150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.253:33539 every 8 connection(s)
I20260923 18:11:12.960302 2082 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
I20260923 18:11:12.963177 2151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:12.967324 2151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:11:12.968456 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 2082
Sep 23 18:11:12 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187072, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.253@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
Sep 23 18:11:12 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.75.253@KRBTEST.COM
I20260923 18:11:12.981483 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.969433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.27.75.254:43469)
0923 18:11:12.969567 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:12.969571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:12.970676 (+ 1105us) server_negotiation.cc:407] Connection header received
0923 18:11:12.971523 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:12.971525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:12.971576 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:12.971650 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:12.972819 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.973537 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.974646 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.974876 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.974970 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.977697 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:12.977701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:12.977704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:12.977741 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:12.979339 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.979879 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.979882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.979884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.979972 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.980306 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.980310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.980311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.980558 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:12.980648 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:12.981154 (+ 506us) server_negotiation.cc:299] Negotiation successful
0923 18:11:12.981302 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:11:12.982510 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.969897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39278)
0923 18:11:12.970506 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:12.970564 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:12.971295 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:12.971724 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:12.971730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:12.972209 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:12.972526 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:12.972549 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.973687 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.973691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:12.974502 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:12.974510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.975041 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.975046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:12.975197 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.975795 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:12.975812 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:12.977505 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:12.979453 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:12.979463 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:12.979476 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:12.979747 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:12.980075 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:12.980077 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:12.980078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:12.980213 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:12.980660 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:12.980667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:12.980908 (+ 241us) client_negotiation.cc:769] Sending connection context
0923 18:11:12.981069 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 18:11:12.981679 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":2,"spinlock_wait_cycles":31104,"thread_start_us":168,"threads_started":1}
I20260923 18:11:12.982509 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.969897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.27.75.253:33539)
0923 18:11:12.970414 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:12.970438 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:12.970481 (+ 43us) server_negotiation.cc:407] Connection header received
0923 18:11:12.971285 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:12.971312 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:12.971581 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:12.971760 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:12.972468 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.973580 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.974500 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.974925 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.975084 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.978134 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:12.978140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:12.978152 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:12.978177 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:12.979574 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.980143 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.980149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.980150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.980225 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.980495 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.980499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.980501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.980919 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:12.981152 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:12.981366 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 18:11:12.981839 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":354,"spinlock_wait_cycles":2304,"thread_start_us":146,"threads_started":1}
I20260923 18:11:12.983842 2151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:11:12 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187072, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.253@KRBTEST.COM for oryx/127.27.75.253@KRBTEST.COM
I20260923 18:11:12.991443 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.984332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:33539 (local address 127.0.0.1:36024)
0923 18:11:12.984681 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:12.984694 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:12.984771 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:12.984953 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:12.984956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:12.985144 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:12.985245 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:12.985251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.985986 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.985989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:12.986505 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:12.986512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.986924 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.986927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:12.987010 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.987424 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:12.987433 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:12.988602 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:12.990146 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:12.990149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:12.990151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:12.990325 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:12.990550 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:12.990553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:12.990555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:12.990625 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:12.990947 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:12.990950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:12.991033 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 18:11:12.991113 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 18:11:12.991266 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 18:11:12.991443 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:12.984388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.27.75.253:33539)
0923 18:11:12.984537 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:12.984541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:12.984744 (+ 203us) server_negotiation.cc:407] Connection header received
0923 18:11:12.984837 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:12.984840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:12.984874 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:12.984935 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:12.985344 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.985899 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.986602 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:12.986834 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:12.986917 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:12.988723 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:12.988727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:12.988729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:12.988753 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:12.990041 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.990404 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.990407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.990409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.990476 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:12.990697 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:12.990700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:12.990702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:12.990867 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:12.990948 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:12.991165 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 18:11:12.991315 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":90}
I20260923 18:11:12.994993 2151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: Bootstrap starting.
I20260923 18:11:12.997359 2151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:12.998085 2151 log.cc:824] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:12.999996 2151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: No bootstrap required, opened a new log
I20260923 18:11:13.003084 2151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } }
I20260923 18:11:13.003327 2151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:13.003366 2151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761778be5f0f4620ad861bb0b32a7762, State: Initialized, Role: FOLLOWER
I20260923 18:11:13.003808 2151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } }
I20260923 18:11:13.004333 2157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } } }
I20260923 18:11:13.004465 2157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:11:13.005031 2151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 18:11:13.009676 2168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:11:13.009773 2168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:11:13.011983 2082 master_runner.cc:428] Detected that this master 761778be5f0f4620ad861bb0b32a7762 is joining an existing cluster
I20260923 18:11:13.012054 2082 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.75.253:33539
I20260923 18:11:13.026841 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.017611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.27.75.254:43469)
0923 18:11:13.017748 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.017752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.018051 (+ 299us) server_negotiation.cc:407] Connection header received
0923 18:11:13.018263 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.018266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.018320 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.018376 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.019136 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.019947 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.021078 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.021485 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.021561 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.023030 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.023034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.023036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.023063 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.024776 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.025483 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.025487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.025489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.025587 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.025922 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.025925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.025927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.026119 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.026180 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.026563 (+ 383us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.026682 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:11:13.026841 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.018242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.27.75.253:33539)
0923 18:11:13.018529 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.018533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.018795 (+ 262us) server_negotiation.cc:407] Connection header received
0923 18:11:13.018926 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.018929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.018960 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.019003 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.019936 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.020655 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.021676 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.021970 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.022036 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.023010 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.023013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.023015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.023035 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.024926 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.025483 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.025487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.025489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.025587 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.025865 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.025869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.025870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.026083 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.026168 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.026544 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.026677 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260923 18:11:13.027379 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.017519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39286)
0923 18:11:13.017971 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.017984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.018132 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.018635 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.018638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.018861 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.019011 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.019018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.020068 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.020071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.020956 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.020963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.021740 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.021743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.021829 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.022389 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.022403 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.022880 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.024925 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.024928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.024930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.025273 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.025744 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.025746 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.025748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.025829 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.026240 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.026243 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.026377 (+ 134us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.027058 (+ 681us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.027208 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":36,"spinlock_wait_cycles":6144,"thread_start_us":120,"threads_started":1}
I20260923 18:11:13.027545 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.018362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:33539 (local address 127.0.0.1:36038)
0923 18:11:13.018725 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.018737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.018846 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.019115 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.019118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.019308 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.019821 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.019831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.020787 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.020790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.021572 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.021581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.022149 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.022151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.022216 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.022539 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.022551 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.022896 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.025022 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.025024 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.025027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.025283 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.025688 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.025691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.025693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.025778 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.026231 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.026233 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.026377 (+ 144us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.027261 (+ 884us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.027370 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":20,"spinlock_wait_cycles":2688,"thread_start_us":83,"threads_started":1}
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187071, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.254@KRBTEST.COM for oryx/127.27.75.253@KRBTEST.COM
I20260923 18:11:13.041518 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.031972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.27.75.253:33539)
0923 18:11:13.032077 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.032081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.032429 (+ 348us) server_negotiation.cc:407] Connection header received
0923 18:11:13.032597 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.032600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.032648 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.032711 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.033382 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.033882 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.034727 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.034951 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.035044 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.037772 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.037776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.037779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.037806 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.039461 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.040011 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.040014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.040016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.040106 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.040402 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.040406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.040407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.040650 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.040722 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.041212 (+ 490us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.041345 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 18:11:13.041540 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.031952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:33539 (local address 127.0.0.1:36040)
0923 18:11:13.032304 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.032344 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.032473 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.032760 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.032766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.033115 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.033279 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.033286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.033984 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.033987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.034612 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.034621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.035072 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.035076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.035216 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.035776 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.035791 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.037633 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.039568 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.039580 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.039587 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.039909 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.040191 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.040194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.040196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.040292 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.040806 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.040813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.040948 (+ 135us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.041099 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.041352 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
I20260923 18:11:13.043447 1612 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.75.253:33539: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "761778be5f0f4620ad861bb0b32a7762"
member_type: NON_VOTER
last_known_addr {
host: "127.27.75.253"
port: 33539
}
attrs {
promote: true
}
}
dest_uuid: "e3a6b8ab2c1340c8a837f723566263c0"
cas_config_opid_index: -1
I20260923 18:11:13.044252 1612 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.27.75.252@KRBTEST.COM; defaulting to no policy
I20260923 18:11:13.045959 2179 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 761778be5f0f4620ad861bb0b32a7762 (127.27.75.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } }
I20260923 18:11:13.046926 2125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:13.048115 2179 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.75.253:33539
I20260923 18:11:13.048399 2125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 FOLLOWER]: Refusing update from remote peer e3a6b8ab2c1340c8a837f723566263c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.252@KRBTEST.COM" created.
I20260923 18:11:13.048592 2082 master.cc:537] Master@127.27.75.253:33539 shutting down...
I20260923 18:11:13.050084 2178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 18:11:13.052201 1605 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 18:11:13.053884 1605 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 -> Peer 761778be5f0f4620ad861bb0b32a7762 (127.27.75.253:33539): Couldn't send request to peer 761778be5f0f4620ad861bb0b32a7762. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 18:11:13.055053 2082 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:13.055006 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } }
I20260923 18:11:13.055131 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:13.055516 2082 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:13.055697 2082 tablet_replica.cc:330] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: stopping tablet replica
I20260923 18:11:13.072219 2082 master.cc:552] Master@127.27.75.253:33539 shutdown complete.
I20260923 18:11:13.072293 2082 master_runner.cc:305] Clearing existing system tablet
I20260923 18:11:13.072541 2082 fs_manager.cc:359] Metadata directory not provided
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.252.keytab.
Entry for principal oryx/127.27.75.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.252.keytab.
I20260923 18:11:13.072597 2082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:13.073493 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.252:40905
--webserver_interface=127.27.75.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.252.keytab
--principal=oryx/127.27.75.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:43469,127.27.75.253:33539,127.27.75.252:40905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:11:13.074153 2082 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 18:11:13.075057 2187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.075264 2082 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:13.075394 2082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 18:11:13.075434 2082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/wal
uuid: "761778be5f0f4620ad861bb0b32a7762"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "b4d6199eb82fae9729f432e514f7d54d"
server_key_iv: "827200d1a789f76d85da55cb77e4a63d"
server_key_version: "encryptionkey@0"
I20260923 18:11:13.077651 2082 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:13.080137 2082 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 18:11:13.080236 2082 log.cc:1196] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 18:11:13.080484 2082 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: Deleting consensus metadata
I20260923 18:11:13.080685 2082 master_runner.cc:315] Copying system tablet from 127.27.75.254:43469
I20260923 18:11:13.081948 2082 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: tablet copy: Beginning tablet copy session from remote peer at address 127.27.75.254:43469
I20260923 18:11:13.091306 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.083015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.27.75.254:43469)
0923 18:11:13.083140 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.083145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.083163 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:11:13.083215 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.083218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.083273 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.083379 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.083892 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.084780 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.085984 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.086346 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.086450 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.087776 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.087780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.087783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.087813 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.089442 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.089994 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.089998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.090000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.090095 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.090402 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.090405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.090407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.090632 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.090733 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.091013 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.091139 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:11:13.091334 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.082569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39290)
0923 18:11:13.082870 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.082887 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.083002 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.083404 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.083407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.083638 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.083782 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.083791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.084894 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.084898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.085841 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.085854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.086476 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.086480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.086580 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.087177 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.087192 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.087609 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.089608 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.089612 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.089614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.089883 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.090205 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.090208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.090211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.090316 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.090732 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.090735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.090848 (+ 113us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.090955 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.091121 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
I20260923 18:11:13.092433 1638 tablet_copy_service.cc:140] P e3a6b8ab2c1340c8a837f723566263c0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 761778be5f0f4620ad861bb0b32a7762 ({username='oryx', principal='oryx/127.27.75.253@KRBTEST.COM'} at 127.0.0.1:39290)
I20260923 18:11:13.092645 1638 tablet_copy_service.cc:161] P e3a6b8ab2c1340c8a837f723566263c0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 761778be5f0f4620ad861bb0b32a7762 at {username='oryx', principal='oryx/127.27.75.253@KRBTEST.COM'} at 127.0.0.1:39290: session id = 761778be5f0f4620ad861bb0b32a7762-00000000000000000000000000000000
I20260923 18:11:13.094801 1638 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 18:11:13.096719 2082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 18:11:13.099248 2082 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: tablet copy: Starting download of 0 data blocks...
I20260923 18:11:13.099462 2082 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: tablet copy: Starting download of 1 WAL segments...
I20260923 18:11:13.101393 2082 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 18:11:13.103428 1638 tablet_copy_service.cc:342] P e3a6b8ab2c1340c8a837f723566263c0: Request end of tablet copy session 761778be5f0f4620ad861bb0b32a7762-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.75.253@KRBTEST.COM'} at 127.0.0.1:39290
I20260923 18:11:13.103583 1638 tablet_copy_service.cc:433] P e3a6b8ab2c1340c8a837f723566263c0: ending tablet copy session 761778be5f0f4620ad861bb0b32a7762-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 761778be5f0f4620ad861bb0b32a7762
I20260923 18:11:13.111258 2082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:13.113168 2194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:13.118263 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:13.118546 2195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:13.118724 2082 server_base.cc:1034] running on GCE node
I20260923 18:11:13.118927 2082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:13.119246 2082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:13.120383 2082 hybrid_clock.cc:648] HybridClock initialized: now 1790187073120357 us; error 42 us; skew 500 ppm
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:13.121995 2082 init.cc:377] Logged in from keytab as oryx/127.27.75.253@KRBTEST.COM (short username oryx)
I20260923 18:11:13.122483 2082 webserver.cc:460] Webserver started at http://127.27.75.253:41739/ using document root <none> and password file <none>
I20260923 18:11:13.122687 2082 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:13.122728 2082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:13.124184 2082 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:13.125330 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.125602 2082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:13.125682 2082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/wal
uuid: "761778be5f0f4620ad861bb0b32a7762"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "b4d6199eb82fae9729f432e514f7d54d"
server_key_iv: "827200d1a789f76d85da55cb77e4a63d"
server_key_version: "encryptionkey@0"
I20260923 18:11:13.125772 2082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:13.142961 2082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:13.145223 2082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:13.150394 2082 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.253:33539
I20260923 18:11:13.150432 2256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.253:33539 every 8 connection(s)
I20260923 18:11:13.151037 2082 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
I20260923 18:11:13.152856 2257 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:11:13.154021 2257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: Bootstrap starting.
I20260923 18:11:13.165221 2257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:11:13.165570 2257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: Bootstrap complete.
I20260923 18:11:13.166236 2257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } }
I20260923 18:11:13.166381 2257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 761778be5f0f4620ad861bb0b32a7762, State: Initialized, Role: LEARNER
I20260923 18:11:13.166493 2257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } }
I20260923 18:11:13.166868 2260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } }
I20260923 18:11:13.166996 2257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:13.167089 2260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: This master's current role is: LEARNER
I20260923 18:11:13.169941 2271 catalog_manager.cc:1259] Loaded cluster ID: af16d14630074fcb8a236c9641704d69
I20260923 18:11:13.170075 2271 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: loading cluster ID for follower catalog manager: success
I20260923 18:11:13.172230 2271 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: acquiring CA information for follower catalog manager: success
I20260923 18:11:13.173697 2271 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 18:11:13.204919 2186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:13.205245 2186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:13.205305 2186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:13.209416 2186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:13.209481 2186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:13.209509 2186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:13.209532 2186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:13.209559 2186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:13.214987 2186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:43469,127.27.75.253:33539,127.27.75.252:40905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.252.keytab
--principal=oryx/127.27.75.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.252:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.75.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:13.216084 2186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:13.217092 2186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:13.225977 2277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:13.226019 2186 server_base.cc:1034] running on GCE node
W20260923 18:11:13.225977 2276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:13.225975 2279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:13.226897 2186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:13.228037 2186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:13.229185 2186 hybrid_clock.cc:648] HybridClock initialized: now 1790187073229158 us; error 49 us; skew 500 ppm
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:13.232111 2186 init.cc:377] Logged in from keytab as oryx/127.27.75.252@KRBTEST.COM (short username oryx)
I20260923 18:11:13.233047 2186 webserver.cc:460] Webserver started at http://127.27.75.252:44957/ using document root <none> and password file <none>
I20260923 18:11:13.233633 2186 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:13.233704 2186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:13.233916 2186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:13.235968 2186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data/instance:
uuid: "a81c8202ca7a474bb59838f9d5902944"
format_stamp: "Formatted at 2026-09-23 18:11:13 on dist-test-slave-wsx5"
server_key: "05413a94482ef654e7875903b4ad06e1"
server_key_iv: "90fdedc6965d645170bd9cb422399705"
server_key_version: "encryptionkey@0"
I20260923 18:11:13.236516 2186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/wal/instance:
uuid: "a81c8202ca7a474bb59838f9d5902944"
format_stamp: "Formatted at 2026-09-23 18:11:13 on dist-test-slave-wsx5"
server_key: "05413a94482ef654e7875903b4ad06e1"
server_key_iv: "90fdedc6965d645170bd9cb422399705"
server_key_version: "encryptionkey@0"
I20260923 18:11:13.240451 2186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:11:13.242946 2286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.244278 2186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:11:13.244427 2186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/wal
uuid: "a81c8202ca7a474bb59838f9d5902944"
format_stamp: "Formatted at 2026-09-23 18:11:13 on dist-test-slave-wsx5"
server_key: "05413a94482ef654e7875903b4ad06e1"
server_key_iv: "90fdedc6965d645170bd9cb422399705"
server_key_version: "encryptionkey@0"
I20260923 18:11:13.244536 2186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:13.259938 2186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:13.263152 2186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:13.263329 2186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:13.270592 2186 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.252:40905
I20260923 18:11:13.270848 2338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.252:40905 every 8 connection(s)
I20260923 18:11:13.271588 2186 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
I20260923 18:11:13.274362 2339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:13.278023 2339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:11:13.280241 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 2186
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.252@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.75.252@KRBTEST.COM
I20260923 18:11:13.292958 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.280155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.27.75.254:43469)
0923 18:11:13.280474 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.280479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.281681 (+ 1202us) server_negotiation.cc:407] Connection header received
0923 18:11:13.282593 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.282597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.282653 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.282720 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.283731 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.284566 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.285949 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.286289 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.286403 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.288984 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.288987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.288989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.289015 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.290599 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.291233 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.291236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.291237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.291352 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.291695 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.291698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.291699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.291923 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.292019 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.292656 (+ 637us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.292786 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:11:13.293964 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.280669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39294)
0923 18:11:13.281538 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.281589 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.282392 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.282778 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.282786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.283243 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.283591 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.283610 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.284702 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.284705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.285816 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.285826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.286448 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.286451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.286595 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.287193 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.287214 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.288841 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.290722 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.290730 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.290740 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.291111 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.291446 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.291448 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.291449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.291587 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.292044 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.292051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.292422 (+ 371us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.292588 (+ 166us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.292924 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":1,"thread_start_us":138,"threads_started":1}
I20260923 18:11:13.293964 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.281001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.27.75.252:40905)
0923 18:11:13.281743 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.281753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.281792 (+ 39us) server_negotiation.cc:407] Connection header received
0923 18:11:13.282410 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.282435 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.282681 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.282929 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.283738 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.284955 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.286081 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.286627 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.286706 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.289495 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.289501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.289510 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.289532 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.291199 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.291627 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.291633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.291635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.291745 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.292044 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.292047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.292049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.292423 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.292664 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.292808 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.292971 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":650,"spinlock_wait_cycles":1408,"thread_start_us":158,"threads_started":1}
I20260923 18:11:13.296682 2339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.252@KRBTEST.COM for oryx/127.27.75.253@KRBTEST.COM
I20260923 18:11:13.304258 1809 heartbeater.cc:503] Master 127.27.75.254:43469 was elected leader, sending a full tablet report...
I20260923 18:11:13.306058 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.297408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:33539 (local address 127.0.0.1:36044)
0923 18:11:13.297572 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.297586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.297664 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.298071 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.298074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.298236 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.298379 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.298385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.299403 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.299406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.299986 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.299996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.300441 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.300443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.300522 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.301056 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.301071 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.302498 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.304424 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.304428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.304430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.304686 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.305029 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.305032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.305034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.305163 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.305579 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.305582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.305692 (+ 110us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.305800 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.305932 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 18:11:13.306166 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.297567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.27.75.253:33539)
0923 18:11:13.297841 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.297845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.297858 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:13.297897 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.297899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.297938 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.298015 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.298521 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.299267 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.300114 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.300348 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.300431 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.302678 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.302681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.302683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.302710 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.304296 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.304796 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.304800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.304802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.304934 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.305262 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.305265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.305266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.305476 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.305571 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.305874 (+ 303us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.306016 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":115,"threads_started":1}
I20260923 18:11:13.306842 2339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.252@KRBTEST.COM for oryx/127.27.75.252@KRBTEST.COM
I20260923 18:11:13.314214 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.307268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:40905 (local address 127.0.0.1:34404)
0923 18:11:13.307381 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.307396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.307463 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.307633 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.307636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.307767 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.307842 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.307846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.308584 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.308587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.309103 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.309111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.309615 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.309619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.309682 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.310080 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.310090 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.311311 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.312823 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.312825 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.312827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.313021 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.313306 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.313312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.313314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.313399 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.313765 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.313767 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.313844 (+ 77us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.313923 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.314034 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 18:11:13.314241 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.307330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.27.75.252:40905)
0923 18:11:13.307432 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.307436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.307447 (+ 11us) server_negotiation.cc:407] Connection header received
0923 18:11:13.307520 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.307522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.307555 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.307612 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.307972 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.308484 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.309227 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.309475 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.309600 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.311432 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.311435 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.311437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.311457 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.312732 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.313103 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.313106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.313107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.313215 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.313495 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.313497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.313508 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.313689 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.313767 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.313971 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.314089 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":35}
I20260923 18:11:13.317572 2339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: Bootstrap starting.
I20260923 18:11:13.319903 2339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:13.320638 2339 log.cc:824] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:13.322654 2339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: No bootstrap required, opened a new log
I20260923 18:11:13.325708 2339 raft_consensus.cc:348] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } }
I20260923 18:11:13.325927 2339 raft_consensus.cc:374] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:13.325986 2339 raft_consensus.cc:729] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a81c8202ca7a474bb59838f9d5902944, State: Initialized, Role: FOLLOWER
I20260923 18:11:13.326428 2339 consensus_queue.cc:260] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } }
I20260923 18:11:13.326996 2345 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } } }
I20260923 18:11:13.327157 2345 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:11:13.327881 2339 sys_catalog.cc:564] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 18:11:13.332923 2356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:11:13.333014 2356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:11:13.337378 2186 master_runner.cc:428] Detected that this master a81c8202ca7a474bb59838f9d5902944 is joining an existing cluster
I20260923 18:11:13.337468 2186 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.75.252:40905
I20260923 18:11:13.347633 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 1680
I20260923 18:11:13.350140 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.340997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.27.75.252:40905)
0923 18:11:13.341082 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.341086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.342021 (+ 935us) server_negotiation.cc:407] Connection header received
0923 18:11:13.342162 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.342165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.342197 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.342237 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.342723 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.343486 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.344512 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.344750 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.344801 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.345902 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.345905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.345908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.345931 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.347614 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.348754 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.348758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.348760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.348895 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.349235 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.349238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.349240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.349448 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.349535 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.349847 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.349998 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":40}
I20260923 18:11:13.350155 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.341204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:40905 (local address 127.0.0.1:34414)
0923 18:11:13.341959 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.341971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.342073 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.342329 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.342331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.342522 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.342616 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.342622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.343590 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.343593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.344412 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.344419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.344920 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.344923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.345013 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.345416 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.345427 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.345784 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.348341 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.348344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.348347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.348654 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.349001 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.349003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.349004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.349090 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.349535 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.349538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.349678 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.349768 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.349934 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":32,"thread_start_us":87,"threads_started":1}
I20260923 18:11:13.355443 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:44753
--local_ip_for_outbound_sockets=127.27.75.193
--tserver_master_addrs=127.27.75.254:43469
--webserver_port=33945
--webserver_interface=127.27.75.193
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab
--principal=oryx/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.75.254:43469,127.27.75.253:33539
--tserver_master_addrs=127.27.75.254:43469,127.27.75.253:33539,127.27.75.252:40905 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:11:13.369583 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.340581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:33539 (local address 127.0.0.1:36060)
0923 18:11:13.340994 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.341006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.341178 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.341526 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.341528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.341686 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.341772 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.341777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.342741 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.342744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.343627 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.343646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.344315 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.344319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.344413 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.344956 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.344970 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.365611 (+ 20641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.367729 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.367732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.367734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.367995 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.368326 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.368329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.368331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.368421 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.368953 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.368956 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.369056 (+ 100us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.369184 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.369363 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":7,"thread_start_us":131,"threads_started":1}
I20260923 18:11:13.370030 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.340118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39304)
0923 18:11:13.340453 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.340468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.340634 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.341030 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.341033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.341297 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.341450 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.341458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.342563 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.342566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.343532 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.343543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.344287 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.344291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.344439 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.365659 (+ 21220us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.365677 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.366126 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.368361 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.368365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.368367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.368615 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.368906 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.368909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.368911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.369003 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.369485 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.369488 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.369600 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.369694 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.369839 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":26,"thread_start_us":111,"threads_started":1}
I20260923 18:11:13.370512 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.340247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.27.75.254:43469)
0923 18:11:13.340385 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.340390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.340552 (+ 162us) server_negotiation.cc:407] Connection header received
0923 18:11:13.340768 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.340772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.340827 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.340880 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.341593 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.342423 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.343674 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.344038 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.344114 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.366329 (+ 22215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.366335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.366337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.366373 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.368209 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.368724 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.368728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.368730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.368812 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.369102 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.369105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.369107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.369357 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.369443 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.370237 (+ 794us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.370359 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 18:11:13.370929 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.340697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.27.75.253:33539)
0923 18:11:13.340818 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.340822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.341078 (+ 256us) server_negotiation.cc:407] Connection header received
0923 18:11:13.341298 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.341302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.341354 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.341407 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.341886 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.342620 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.343752 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.344107 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.344199 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.365849 (+ 21650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.365853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.365856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.365889 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.367530 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.368133 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.368136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.368138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.368228 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.368516 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.368519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.368521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.368825 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.368911 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.370416 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.370556 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187071, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.254@KRBTEST.COM for oryx/127.27.75.252@KRBTEST.COM
I20260923 18:11:13.383772 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.374704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:40905 (local address 127.0.0.1:34430)
0923 18:11:13.374816 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.374830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.374903 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.375168 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.375171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.375359 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.375468 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.375475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.376447 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.376451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.377493 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.377507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.378042 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.378046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.378150 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.378631 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.378646 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.380191 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.382033 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.382036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.382039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.382293 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.382577 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.382580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.382582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.382677 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.383184 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.383188 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.383328 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.383422 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.383589 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 18:11:13.384596 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.374772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.27.75.252:40905)
0923 18:11:13.374882 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.374887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.374900 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:13.375001 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.375004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.375057 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.375134 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.375572 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.376343 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.377630 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.377930 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.378027 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.380331 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.380335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.380339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.380366 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.381915 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.382395 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.382399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.382401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.382485 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.382764 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.382767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.382769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.383047 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.383139 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.384253 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.384425 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 18:11:13.385730 1612 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.75.252:40905: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a81c8202ca7a474bb59838f9d5902944"
member_type: NON_VOTER
last_known_addr {
host: "127.27.75.252"
port: 40905
}
attrs {
promote: true
}
}
dest_uuid: "e3a6b8ab2c1340c8a837f723566263c0"
cas_config_opid_index: 5
I20260923 18:11:13.386343 1612 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } }
I20260923 18:11:13.387385 2180 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a81c8202ca7a474bb59838f9d5902944 (127.27.75.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.388052 2313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:13.388994 2180 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.75.252:40905
I20260923 18:11:13.389209 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.389314 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:13.389567 2313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 FOLLOWER]: Refusing update from remote peer e3a6b8ab2c1340c8a837f723566263c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 18:11:13.390272 2179 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:13.392584 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.387774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:33539 (local address 127.0.0.1:36068)
0923 18:11:13.388210 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.388224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.388300 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.388715 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.388718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.388736 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:13.388821 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.388827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.389879 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.389882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.391091 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.391103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.392161 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.392165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.392254 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.392302 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.392370 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.392443 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367}
I20260923 18:11:13.392961 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.388011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.27.75.253:33539)
0923 18:11:13.388125 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.388129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.388371 (+ 242us) server_negotiation.cc:407] Connection header received
0923 18:11:13.388423 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.388426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.388485 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.388583 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:13.388950 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.389757 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.391236 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.391857 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.391955 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.392100 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.392774 (+ 674us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.392851 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 18:11:13.393384 2231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 LEARNER]: Refusing update from remote peer e3a6b8ab2c1340c8a837f723566263c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 18:11:13.393832 2179 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 18:11:13.394373 2186 master.cc:537] Master@127.27.75.252:40905 shutting down...
I20260923 18:11:13.395850 2231 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a81c8202ca7a474bb59838f9d5902944 (127.27.75.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.398928 2313 raft_consensus.cc:858] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } }
W20260923 18:11:13.399145 2313 raft_consensus.cc:1565] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } }
I20260923 18:11:13.399233 2260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3a6b8ab2c1340c8a837f723566263c0. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.399365 2260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: This master's current role is: LEARNER
I20260923 18:11:13.399466 2260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.399546 2260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: This master's current role is: LEARNER
I20260923 18:11:13.400759 2313 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 761778be5f0f4620ad861bb0b32a7762 (127.27.75.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } }
I20260923 18:11:13.403038 2345 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3a6b8ab2c1340c8a837f723566263c0. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } }
I20260923 18:11:13.403149 2345 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 18:11:13.403237 2345 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } }
I20260923 18:11:13.403291 2345 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 18:11:13.403347 2345 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } }
I20260923 18:11:13.403394 2345 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 18:11:13.404621 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: NON_VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: true } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.404773 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:13.405484 2186 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260923 18:11:13.405711 2186 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260923 18:11:13.405799 2186 tablet_replica.cc:330] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: stopping tablet replica
W20260923 18:11:13.406363 1605 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 -> Peer a81c8202ca7a474bb59838f9d5902944 (127.27.75.252:40905): Couldn't send request to peer a81c8202ca7a474bb59838f9d5902944. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 18:11:13.407481 2178 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0: attempting to promote NON_VOTER 761778be5f0f4620ad861bb0b32a7762 to VOTER
I20260923 18:11:13.407972 2178 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } }
W20260923 18:11:13.409722 1605 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 -> Peer a81c8202ca7a474bb59838f9d5902944 (127.27.75.252:40905): Couldn't send request to peer a81c8202ca7a474bb59838f9d5902944. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 18:11:13.409724 2231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 LEARNER]: Refusing update from remote peer e3a6b8ab2c1340c8a837f723566263c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 18:11:13.410131 2183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 18:11:13.411794 2183 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 761778be5f0f4620ad861bb0b32a7762 (127.27.75.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.413424 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.413558 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:13.413703 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.413785 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:13.413770 2231 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 761778be5f0f4620ad861bb0b32a7762 (127.27.75.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.415190 2368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.415315 2368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:11:13.424510 2186 master.cc:552] Master@127.27.75.252:40905 shutdown complete.
I20260923 18:11:13.424587 2186 master_runner.cc:305] Clearing existing system tablet
I20260923 18:11:13.424820 2186 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:13.424876 2186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:13.426307 2186 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:13.427237 2376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.427487 2186 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:13.427793 2186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:13.427845 2186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/wal
uuid: "a81c8202ca7a474bb59838f9d5902944"
format_stamp: "Formatted at 2026-09-23 18:11:13 on dist-test-slave-wsx5"
server_key: "05413a94482ef654e7875903b4ad06e1"
server_key_iv: "90fdedc6965d645170bd9cb422399705"
server_key_version: "encryptionkey@0"
I20260923 18:11:13.429836 2186 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:13.432214 2186 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 18:11:13.432312 2186 log.cc:1196] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 18:11:13.432591 2186 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: Deleting consensus metadata
I20260923 18:11:13.432760 2186 master_runner.cc:315] Copying system tablet from 127.27.75.254:43469
I20260923 18:11:13.433997 2186 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: tablet copy: Beginning tablet copy session from remote peer at address 127.27.75.254:43469
I20260923 18:11:13.442960 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.434562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39312)
0923 18:11:13.434880 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.434894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.435014 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.435276 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.435278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.435499 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.435617 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.435623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.436673 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.436676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.437578 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.437590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.438301 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.438305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.438401 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.438926 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.438940 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.439425 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.441401 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.441404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.441407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.441673 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.441946 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.441949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.441950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.442040 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.442429 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.442432 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.442531 (+ 99us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.442639 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.442778 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":84,"threads_started":1}
I20260923 18:11:13.442978 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.434684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.27.75.254:43469)
0923 18:11:13.434824 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.434829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.434978 (+ 149us) server_negotiation.cc:407] Connection header received
0923 18:11:13.435098 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.435101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.435156 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.435253 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.435747 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.436569 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.437798 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.438167 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.438288 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.439583 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.439587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.439591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.439619 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.441299 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.441774 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.441778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.441780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.441867 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.442123 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.442126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.442128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.442339 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.442428 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.442701 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.442824 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 18:11:13.443840 1638 tablet_copy_service.cc:140] P e3a6b8ab2c1340c8a837f723566263c0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a81c8202ca7a474bb59838f9d5902944 ({username='oryx', principal='oryx/127.27.75.252@KRBTEST.COM'} at 127.0.0.1:39312)
I20260923 18:11:13.443950 1638 tablet_copy_service.cc:161] P e3a6b8ab2c1340c8a837f723566263c0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a81c8202ca7a474bb59838f9d5902944 at {username='oryx', principal='oryx/127.27.75.252@KRBTEST.COM'} at 127.0.0.1:39312: session id = a81c8202ca7a474bb59838f9d5902944-00000000000000000000000000000000
I20260923 18:11:13.445034 1638 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 18:11:13.446343 2186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 18:11:13.448812 2186 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: tablet copy: Starting download of 0 data blocks...
I20260923 18:11:13.449000 2186 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: tablet copy: Starting download of 1 WAL segments...
I20260923 18:11:13.450630 2186 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 18:11:13.452551 1638 tablet_copy_service.cc:342] P e3a6b8ab2c1340c8a837f723566263c0: Request end of tablet copy session a81c8202ca7a474bb59838f9d5902944-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.75.252@KRBTEST.COM'} at 127.0.0.1:39312
I20260923 18:11:13.452659 1638 tablet_copy_service.cc:433] P e3a6b8ab2c1340c8a837f723566263c0: ending tablet copy session a81c8202ca7a474bb59838f9d5902944-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a81c8202ca7a474bb59838f9d5902944
I20260923 18:11:13.460680 2186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:13.462600 2383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:13.462615 2384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:13.463256 2386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:13.463289 2186 server_base.cc:1034] running on GCE node
I20260923 18:11:13.463590 2186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:13.463912 2186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:13.465068 2186 hybrid_clock.cc:648] HybridClock initialized: now 1790187073465052 us; error 31 us; skew 500 ppm
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:13.466935 2186 init.cc:377] Logged in from keytab as oryx/127.27.75.252@KRBTEST.COM (short username oryx)
I20260923 18:11:13.467481 2186 webserver.cc:460] Webserver started at http://127.27.75.252:44433/ using document root <none> and password file <none>
I20260923 18:11:13.467680 2186 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:13.467725 2186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:13.469485 2186 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 18:11:13.470561 2393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.470784 2186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 18:11:13.470852 2186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/wal
uuid: "a81c8202ca7a474bb59838f9d5902944"
format_stamp: "Formatted at 2026-09-23 18:11:13 on dist-test-slave-wsx5"
server_key: "05413a94482ef654e7875903b4ad06e1"
server_key_iv: "90fdedc6965d645170bd9cb422399705"
server_key_version: "encryptionkey@0"
I20260923 18:11:13.470939 2186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:13.490499 2186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:13.492605 2186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:13.497702 2186 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.252:40905
I20260923 18:11:13.498356 2186 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
I20260923 18:11:13.501351 2445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.252:40905 every 8 connection(s)
I20260923 18:11:13.503110 2449 sys_catalog.cc:263] Verifying existing consensus state
W20260923 18:11:13.504168 2365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260923 18:11:13.504408 2449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: Bootstrap starting.
W20260923 18:11:13.504483 2365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:13.504554 2365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:13.509392 2365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:13.509469 2365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:13.509549 2365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:13.513345 2449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:11:13.513638 2449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: Bootstrap complete.
I20260923 18:11:13.514127 2449 raft_consensus.cc:348] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } }
I20260923 18:11:13.514245 2449 raft_consensus.cc:729] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a81c8202ca7a474bb59838f9d5902944, State: Initialized, Role: LEARNER
I20260923 18:11:13.514348 2449 consensus_queue.cc:260] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } }
I20260923 18:11:13.514699 2452 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.514777 2449 sys_catalog.cc:564] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:13.514833 2452 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: This master's current role is: LEARNER
I20260923 18:11:13.515488 2365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab
--principal=oryx/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:44753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=33945
--tserver_master_addrs=127.27.75.254:43469,127.27.75.253:33539,127.27.75.252:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:13.516707 2365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:13.517436 2463 catalog_manager.cc:1259] Loaded cluster ID: af16d14630074fcb8a236c9641704d69
I20260923 18:11:13.517482 2463 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: loading cluster ID for follower catalog manager: success
I20260923 18:11:13.518184 2365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:11:13.519189 2463 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: acquiring CA information for follower catalog manager: success
I20260923 18:11:13.520284 2463 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:11:13.523177 1943 heartbeater.cc:503] Master 127.27.75.254:43469 was elected leader, sending a full tablet report...
W20260923 18:11:13.528357 2465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:13.528448 2365 server_base.cc:1034] running on GCE node
W20260923 18:11:13.528383 2468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:13.528352 2466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:13.529026 2365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:13.529696 2365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:13.530851 2365 hybrid_clock.cc:648] HybridClock initialized: now 1790187073530829 us; error 34 us; skew 500 ppm
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:13.533644 2365 init.cc:377] Logged in from keytab as oryx/127.27.75.193@KRBTEST.COM (short username oryx)
I20260923 18:11:13.534525 2365 webserver.cc:460] Webserver started at http://127.27.75.193:33945/ using document root <none> and password file <none>
I20260923 18:11:13.535035 2365 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:13.535104 2365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:13.539084 2365 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:11:13.541146 2475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.542676 2365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:11:13.542780 2365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "5f45b3b090e44171bed8b23947a2bb7a"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "bc74d4820a20b1db431ffe3b0c1db47f"
server_key_iv: "878f6f7ec9dd64751612b62a44c52d92"
server_key_version: "encryptionkey@0"
I20260923 18:11:13.543329 2365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:13.557494 2365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:13.560595 2365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:13.560751 2365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:13.561390 2365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:13.562234 2365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:13.562282 2365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.562321 2365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:13.562341 2365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.571182 2365 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:44753
I20260923 18:11:13.571216 2588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:44753 every 8 connection(s)
I20260923 18:11:13.572221 2365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:13.580312 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 2365
I20260923 18:11:13.580515 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 1814
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.253@KRBTEST.COM
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.252@KRBTEST.COM
I20260923 18:11:13.588110 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:45679
--local_ip_for_outbound_sockets=127.27.75.194
--tserver_master_addrs=127.27.75.254:43469
--webserver_port=32793
--webserver_interface=127.27.75.194
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab
--principal=oryx/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.75.254:43469,127.27.75.253:33539
--tserver_master_addrs=127.27.75.254:43469,127.27.75.253:33539,127.27.75.252:40905 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:11:13.589545 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.575984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:39213 (local address 127.27.75.253:33539)
0923 18:11:13.576108 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.576112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.576123 (+ 11us) server_negotiation.cc:407] Connection header received
0923 18:11:13.576490 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.576493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.576544 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.576611 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.578098 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.578930 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.580143 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.580478 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.580551 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.585373 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.585378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.585381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.585413 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.586929 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.587624 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.587628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.587629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.587735 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.588156 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.588159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.588161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.588397 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.588464 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.589240 (+ 776us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.589367 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 18:11:13.589658 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.574040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:53267 (local address 127.27.75.254:43469)
0923 18:11:13.574183 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.574188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.575055 (+ 867us) server_negotiation.cc:407] Connection header received
0923 18:11:13.576523 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.576526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.576585 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.576638 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.578084 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.578641 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.579826 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.580098 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.580200 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.584718 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.584722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.584725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.584758 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.587225 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.587696 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.587700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.587701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.587789 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.588159 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.588162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.588164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.588387 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.588452 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.589366 (+ 914us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.589501 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:11:13.590631 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.575317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:40905 (local address 127.27.75.193:57077)
0923 18:11:13.575578 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.575591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.576295 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.576843 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.576852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.577369 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.577951 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.577958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.579251 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.579254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.580053 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.580060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.580732 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.580735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.581008 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.581839 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.581869 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.585747 (+ 3878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.587636 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.587639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.587640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.587868 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.588153 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.588156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.588158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.588238 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.588663 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.588666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.588925 (+ 259us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.589190 (+ 265us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.589550 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":294,"spinlock_wait_cycles":59008,"thread_start_us":76,"threads_started":1}
I20260923 18:11:13.591038 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.575487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:57077 (local address 127.27.75.252:40905)
0923 18:11:13.576014 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.576018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.576033 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:11:13.576587 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.576591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.576701 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.576788 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.578098 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.579126 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.580182 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.580537 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.580619 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.585922 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.585926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.585929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.585962 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.587525 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.587966 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.587969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.587971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.588064 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.588333 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.588336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.588338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.588552 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.588627 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.590738 (+ 2111us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.590871 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":83,"threads_started":1}
I20260923 18:11:13.590634 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.574448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.193:53267)
0923 18:11:13.574901 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.574947 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.576294 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.576842 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.576852 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.577370 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.577939 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.577953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.578780 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.578782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.579703 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.579712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.580405 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.580410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.580633 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.581978 (+ 1345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.581993 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.584483 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.587350 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.587353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.587355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.587582 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.587898 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.587901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.587903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.588051 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.589059 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.589062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.589175 (+ 113us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.590194 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.590340 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":58,"thread_start_us":105,"threads_started":1}
I20260923 18:11:13.590633 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.575640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:33539 (local address 127.27.75.193:39213)
0923 18:11:13.575833 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.575847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.576294 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.576843 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.576852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.577369 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.577948 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.577958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.579067 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.579070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.580002 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.580012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.580720 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.580724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.581049 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.581839 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.581869 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.585184 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.587081 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.587090 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.587111 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.587488 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.587847 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.587849 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.587851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.588049 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.588555 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.588562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.588925 (+ 363us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.589693 (+ 768us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.589825 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":102,"spinlock_wait_cycles":62464,"thread_start_us":63,"threads_started":1}
I20260923 18:11:13.592020 2589 heartbeater.cc:347] Connected to a master server at 127.27.75.252:40905
I20260923 18:11:13.592124 2590 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43469
I20260923 18:11:13.592283 2589 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:13.592283 2590 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:13.592597 2591 heartbeater.cc:347] Connected to a master server at 127.27.75.253:33539
I20260923 18:11:13.592658 2591 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:13.592916 2589 heartbeater.cc:511] Master 127.27.75.252:40905 requested a full tablet report, sending...
I20260923 18:11:13.592914 2591 heartbeater.cc:511] Master 127.27.75.253:33539 requested a full tablet report, sending...
I20260923 18:11:13.592914 2590 heartbeater.cc:511] Master 127.27.75.254:43469 requested a full tablet report, sending...
I20260923 18:11:13.594307 1612 ts_manager.cc:194] Re-registered known tserver with Master: 5f45b3b090e44171bed8b23947a2bb7a (127.27.75.193:44753)
I20260923 18:11:13.594475 2221 ts_manager.cc:194] Registered new tserver with Master: 5f45b3b090e44171bed8b23947a2bb7a (127.27.75.193:44753)
I20260923 18:11:13.595036 1612 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:53267
I20260923 18:11:13.595527 2410 ts_manager.cc:194] Registered new tserver with Master: 5f45b3b090e44171bed8b23947a2bb7a (127.27.75.193:44753)
W20260923 18:11:13.714601 2597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:13.714893 2597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:13.714946 2597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:13.719205 2597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:13.719277 2597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:13.719340 2597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:13.724825 2597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab
--principal=oryx/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:45679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=32793
--tserver_master_addrs=127.27.75.254:43469,127.27.75.253:33539,127.27.75.252:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:13.726018 2597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:13.727334 2597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:13.739480 2603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:13.739463 2605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:13.739820 2597 server_base.cc:1034] running on GCE node
W20260923 18:11:13.739463 2602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:13.740468 2597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:13.741189 2597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:13.742388 2597 hybrid_clock.cc:648] HybridClock initialized: now 1790187073742373 us; error 28 us; skew 500 ppm
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:13.745491 2597 init.cc:377] Logged in from keytab as oryx/127.27.75.194@KRBTEST.COM (short username oryx)
I20260923 18:11:13.746450 2597 webserver.cc:460] Webserver started at http://127.27.75.194:32793/ using document root <none> and password file <none>
I20260923 18:11:13.747041 2077 heartbeater.cc:503] Master 127.27.75.254:43469 was elected leader, sending a full tablet report...
I20260923 18:11:13.747064 2597 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:13.747146 2597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:13.751322 2597 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:11:13.753648 2612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.754978 2597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:13.755124 2597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "6cb3fc9dd9234ff3b6e27343823ef836"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "5f7f473b8a67f7d74a01119e2f0f7cfe"
server_key_iv: "6f687fddbcb4d1f01caa4dce26fea5a2"
server_key_version: "encryptionkey@0"
I20260923 18:11:13.755692 2597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:13.771407 2597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:13.774926 2597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:13.775130 2597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:13.775738 2597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:13.776626 2597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:13.776675 2597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.776731 2597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:13.776757 2597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.785189 2597 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:45679
I20260923 18:11:13.785244 2725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:45679 every 8 connection(s)
I20260923 18:11:13.786193 2597 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:13.795852 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 2597
I20260923 18:11:13.796051 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 1948
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.252@KRBTEST.COM
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.253@KRBTEST.COM
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
I20260923 18:11:13.802950 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.787921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:55045 (local address 127.27.75.254:43469)
0923 18:11:13.788086 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.788090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.788938 (+ 848us) server_negotiation.cc:407] Connection header received
0923 18:11:13.790389 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.790393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.790452 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.790520 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.791985 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.792508 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.794089 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.794319 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.794380 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.798758 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.798762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.798765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.798795 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.800294 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.801024 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.801028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.801029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.801154 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.801593 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.801596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.801598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.801815 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.801891 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.802643 (+ 752us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.802779 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:11:13.802950 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.789792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:42269 (local address 127.27.75.252:40905)
0923 18:11:13.789903 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.789908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.789922 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:13.790685 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.790689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.790743 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.790819 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.792003 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.792814 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.794090 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.794447 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.794529 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.798623 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.798627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.798630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.798661 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.800531 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.801046 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.801049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.801050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.801165 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.801599 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.801602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.801604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.801815 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.801887 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.802643 (+ 756us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.802779 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 18:11:13.803225 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:43129
--local_ip_for_outbound_sockets=127.27.75.195
--tserver_master_addrs=127.27.75.254:43469
--webserver_port=34313
--webserver_interface=127.27.75.195
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab
--principal=oryx/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:32859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.75.254:43469,127.27.75.253:33539
--tserver_master_addrs=127.27.75.254:43469,127.27.75.253:33539,127.27.75.252:40905 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:11:13.803884 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.789443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:58569 (local address 127.27.75.253:33539)
0923 18:11:13.789580 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.789584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.789648 (+ 64us) server_negotiation.cc:407] Connection header received
0923 18:11:13.790493 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.790496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.790540 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.790607 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.791995 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.792811 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.794041 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.794375 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.794473 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.798384 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:13.798388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:13.798391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:13.798421 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:13.800416 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.801041 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.801044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.801046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.801177 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:13.801568 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:13.801571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:13.801573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:13.801803 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:13.801888 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.803607 (+ 1719us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.803734 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 18:11:13.804364 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.788346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.194:55045)
0923 18:11:13.788793 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.788834 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.790178 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.790720 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.790730 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.791260 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.791841 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.791855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.792649 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.792652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.793934 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.793945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.794530 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.794534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.794808 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.795709 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.795722 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.798604 (+ 2882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.800434 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.800441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.800460 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.800906 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.801264 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.801267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.801269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.801487 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.801947 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.801950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.802347 (+ 397us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.803158 (+ 811us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.803319 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":242,"spinlock_wait_cycles":50304,"thread_start_us":126,"threads_started":1}
I20260923 18:11:13.804584 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.789622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:40905 (local address 127.27.75.194:42269)
0923 18:11:13.789815 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.789827 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.790190 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.790848 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.790851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.791264 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.791854 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.791860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.792916 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.792919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.793939 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.793945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.794654 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.794657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.794768 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.795553 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.795688 (+ 135us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.798448 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.800656 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.800659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.800661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.800944 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.801264 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.801267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.801269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.801483 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.801947 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.801950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.802347 (+ 397us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.803104 (+ 757us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.803266 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":236,"spinlock_wait_cycles":75648,"thread_start_us":75,"threads_started":1}
I20260923 18:11:13.804911 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.789294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:33539 (local address 127.27.75.194:58569)
0923 18:11:13.789559 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.789573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.790184 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.790721 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.790730 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.791260 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.791851 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.791860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.792942 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.792945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.793931 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.793941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.794530 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.794534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.794832 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.795694 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:13.795712 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:13.798172 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:13.800547 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.800550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.800553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.800908 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.801269 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:13.801273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:13.801275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:13.801469 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:13.801941 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:13.801948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:13.802349 (+ 401us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.802558 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.802916 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":285,"spinlock_wait_cycles":46592,"thread_start_us":74,"threads_started":1}
I20260923 18:11:13.805665 2727 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43469
I20260923 18:11:13.805913 2727 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:13.805950 2726 heartbeater.cc:347] Connected to a master server at 127.27.75.252:40905
I20260923 18:11:13.806010 2726 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:13.806294 2728 heartbeater.cc:347] Connected to a master server at 127.27.75.253:33539
I20260923 18:11:13.806370 2728 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:13.806545 2726 heartbeater.cc:511] Master 127.27.75.252:40905 requested a full tablet report, sending...
I20260923 18:11:13.806547 2728 heartbeater.cc:511] Master 127.27.75.253:33539 requested a full tablet report, sending...
I20260923 18:11:13.806545 2727 heartbeater.cc:511] Master 127.27.75.254:43469 requested a full tablet report, sending...
I20260923 18:11:13.807855 2410 ts_manager.cc:194] Registered new tserver with Master: 6cb3fc9dd9234ff3b6e27343823ef836 (127.27.75.194:45679)
I20260923 18:11:13.807855 1612 ts_manager.cc:194] Re-registered known tserver with Master: 6cb3fc9dd9234ff3b6e27343823ef836 (127.27.75.194:45679)
I20260923 18:11:13.808625 1612 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:55045
I20260923 18:11:13.809235 2221 ts_manager.cc:194] Registered new tserver with Master: 6cb3fc9dd9234ff3b6e27343823ef836 (127.27.75.194:45679)
W20260923 18:11:13.918654 2733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:13.918926 2733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:13.918978 2733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:13.923028 2733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:13.923087 2733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:13.923163 2733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:13.928680 2733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:32859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab
--principal=oryx/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:43129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=34313
--tserver_master_addrs=127.27.75.254:43469,127.27.75.253:33539,127.27.75.252:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:13.929792 2733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:13.930781 2733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:13.940618 2741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:13.940724 2738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:13.940618 2739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:13.940881 2733 server_base.cc:1034] running on GCE node
I20260923 18:11:13.941409 2733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:13.941987 2733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:13.943156 2733 hybrid_clock.cc:648] HybridClock initialized: now 1790187073943128 us; error 36 us; skew 500 ppm
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:13.946336 2733 init.cc:377] Logged in from keytab as oryx/127.27.75.195@KRBTEST.COM (short username oryx)
I20260923 18:11:13.947319 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.943200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:40905 (local address 127.0.0.1:34438)
0923 18:11:13.943444 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.943461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.943631 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.943931 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.943935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.943958 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:13.944072 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.944079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.944950 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.944954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.946026 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.946039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.946958 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.946961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.947032 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.947056 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:11:13.947107 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 18:11:13.947198 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260923 18:11:13.947342 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.943300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.27.75.252:40905)
0923 18:11:13.943463 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.943467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.943538 (+ 71us) server_negotiation.cc:407] Connection header received
0923 18:11:13.943752 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.943755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.943817 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.943895 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:13.944206 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.944837 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.946184 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.946840 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.946952 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:13.947114 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:13.947151 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 18:11:13.947224 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 18:11:13.947487 2733 webserver.cc:460] Webserver started at http://127.27.75.195:34313/ using document root <none> and password file <none>
I20260923 18:11:13.948045 2420 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 18:11:13.948208 2733 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:13.948310 2733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:13.948325 2452 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3a6b8ab2c1340c8a837f723566263c0. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:13.948465 2452 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: This master's current role is: LEARNER
I20260923 18:11:13.952268 2733 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:11:13.954443 2749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.955818 2733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:11:13.955958 2733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "90362291c0604c48ae4d307f55f84b2f"
format_stamp: "Formatted at 2026-09-23 18:11:12 on dist-test-slave-wsx5"
server_key: "4cf226347270dd66ce66a06391031d4a"
server_key_iv: "bff0d3b8e2ac8e8b6f055912bd530369"
server_key_version: "encryptionkey@0"
I20260923 18:11:13.956578 2733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:13.976922 2733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:13.980381 2733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:13.980578 2733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:13.981256 2733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:13.982230 2733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:13.982280 2733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.982321 2733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:13.982347 2733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:13.991050 2733 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:43129
I20260923 18:11:13.991084 2862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:43129 every 8 connection(s)
I20260923 18:11:13.992018 2733 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:14.000803 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 2733
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.252@KRBTEST.COM
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
Sep 23 18:11:13 dist-test-slave-wsx5 krb5kdc[1573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187073, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.253@KRBTEST.COM
I20260923 18:11:14.010344 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.995312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.195:49353)
0923 18:11:13.995508 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.995521 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.995869 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.996380 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.996387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.996859 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.997433 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.997442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.999200 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.999203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.000136 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.000147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.000820 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.000824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.000917 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.001342 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:14.001356 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:14.004561 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:14.007150 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:14.007154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:14.007156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:14.007655 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:14.007958 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:14.007960 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:14.007962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:14.008050 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:14.008475 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:14.008478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:14.008574 (+ 96us) client_negotiation.cc:769] Sending connection context
0923 18:11:14.008802 (+ 228us) client_negotiation.cc:240] Negotiation successful
0923 18:11:14.009656 (+ 854us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":166,"spinlock_wait_cycles":19584,"thread_start_us":70,"threads_started":1}
I20260923 18:11:14.010730 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.995133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:36261 (local address 127.27.75.253:33539)
0923 18:11:13.995270 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.995274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.995291 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:11:13.996071 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.996075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.996132 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.996204 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.997571 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.998401 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.000426 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.000701 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.000799 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.004644 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:14.004648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:14.004651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:14.004678 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:14.006354 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.006997 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.007001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.007003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.007094 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.007801 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.007805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.007807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.008038 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:14.008138 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.010446 (+ 2308us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.010569 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 18:11:14.011906 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.995488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:49353 (local address 127.27.75.254:43469)
0923 18:11:13.995619 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.995623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.995636 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:13.996086 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.996089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.996141 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.996192 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.998542 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.999077 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.000293 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.000599 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.000697 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.004710 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:14.004713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:14.004716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:14.004741 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:14.007042 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.007765 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.007769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.007770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.007861 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.008146 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.008149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.008151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.008349 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:14.008433 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.011624 (+ 3191us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.011760 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 18:11:14.010349 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.994980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:33539 (local address 127.27.75.195:36261)
0923 18:11:13.995236 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.995249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.995873 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.996378 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.996385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.996849 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.997426 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.997441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.999504 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.999507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.000330 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.000337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.001039 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.001043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.001161 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.001520 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:14.001534 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:14.004449 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:14.006492 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:14.006498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:14.006512 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:14.006878 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:14.007215 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:14.007218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:14.007220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:14.007686 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:14.008196 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:14.008204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:14.008577 (+ 373us) client_negotiation.cc:769] Sending connection context
0923 18:11:14.008802 (+ 225us) client_negotiation.cc:240] Negotiation successful
0923 18:11:14.009193 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":21,"spinlock_wait_cycles":563456,"thread_start_us":74,"threads_started":1}
I20260923 18:11:14.012532 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.002816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.27.75.254:43469)
0923 18:11:14.003333 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.003403 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.003417 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:14.003455 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.003458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.003509 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.003595 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:14.004652 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.005232 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.006486 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.006853 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.006954 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.008630 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:14.008634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:14.008637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:14.008659 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:14.010249 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.010952 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.010956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.010957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.011051 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.011370 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.011374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.011375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.011588 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:14.011706 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.012256 (+ 550us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.012381 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":73,"threads_started":1}
I20260923 18:11:14.012775 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.993764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:43983 (local address 127.27.75.252:40905)
0923 18:11:13.993918 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:13.993923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:13.994691 (+ 768us) server_negotiation.cc:407] Connection header received
0923 18:11:13.996085 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:13.996089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:13.996154 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:13.996203 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:13.997596 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.998401 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.999764 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.999980 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.000064 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.005712 (+ 5648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:14.005717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:14.005720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:14.005750 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:14.007228 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.011413 (+ 4185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.011417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.011419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.011508 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.011843 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.011847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.011849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.012073 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:14.012155 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.012500 (+ 345us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.012635 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 18:11:14.013236 2867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:13.994142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:40905 (local address 127.27.75.195:43983)
0923 18:11:13.994556 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:13.994595 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:13.995866 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:13.996380 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:13.996387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:13.996854 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:13.997436 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.997442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:13.998529 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:13.998532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:13.999654 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:13.999663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.000112 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.000117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.000317 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.000999 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:14.001018 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:14.005514 (+ 4496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:14.007332 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:14.007334 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:14.007336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:14.011273 (+ 3937us) spinlock_profiling.cc:243] Waited 3.34 ms on lock 0x33d586b8. stack: 00007f3fde094c78 00007f3fde0951a9 00007f3fddc571d7 00000000005984a6 00000000005984f9 000000000059c488 00007f3fde0912ab 00007f3fde090ad0 00007f3fe27690ee 00007f3fe26eccbe 00007f3fe26ec2c8 00007f3fe26eb146 00007f3fe26e7e17 00007f3fe2728832 00007f3fe272968c 00007f3fe273f0d3
0923 18:11:14.011298 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:14.011641 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:14.011644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:14.011646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:14.011735 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:14.012212 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:14.012215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:14.012304 (+ 89us) client_negotiation.cc:769] Sending connection context
0923 18:11:14.012893 (+ 589us) client_negotiation.cc:240] Negotiation successful
0923 18:11:14.013040 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":154,"spinlock_wait_cycles":7712512,"thread_start_us":105,"threads_started":1}
I20260923 18:11:14.013376 2864 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43469
I20260923 18:11:14.013664 2864 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:14.014299 2864 heartbeater.cc:511] Master 127.27.75.254:43469 requested a full tablet report, sending...
I20260923 18:11:14.015337 2865 heartbeater.cc:347] Connected to a master server at 127.27.75.253:33539
I20260923 18:11:14.015432 2865 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:14.015648 2865 heartbeater.cc:511] Master 127.27.75.253:33539 requested a full tablet report, sending...
I20260923 18:11:14.015831 2863 heartbeater.cc:347] Connected to a master server at 127.27.75.252:40905
I20260923 18:11:14.015883 2863 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:14.015861 1612 ts_manager.cc:194] Re-registered known tserver with Master: 90362291c0604c48ae4d307f55f84b2f (127.27.75.195:43129)
I20260923 18:11:14.016024 2863 heartbeater.cc:511] Master 127.27.75.252:40905 requested a full tablet report, sending...
I20260923 18:11:14.016533 1612 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:49353
I20260923 18:11:14.016746 2410 ts_manager.cc:194] Registered new tserver with Master: 90362291c0604c48ae4d307f55f84b2f (127.27.75.195:43129)
I20260923 18:11:14.017191 2221 ts_manager.cc:194] Registered new tserver with Master: 90362291c0604c48ae4d307f55f84b2f (127.27.75.195:43129)
I20260923 18:11:14.019366 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.003278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.27.75.253:33539)
0923 18:11:14.003486 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.003490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.003746 (+ 256us) server_negotiation.cc:407] Connection header received
0923 18:11:14.003899 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.003902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.003950 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.004008 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:14.007939 (+ 3931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.008698 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.009825 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.010153 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.010224 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.011817 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:14.011821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:14.011824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:14.011848 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:14.014941 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.017936 (+ 2995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.017940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.017942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.018044 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.018367 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.018370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.018372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.018603 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:14.018733 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.019060 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.019195 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
I20260923 18:11:14.026790 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.004277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.27.75.252:40905)
0923 18:11:14.010787 (+ 6510us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.017926 (+ 7139us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.017945 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:11:14.017983 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.017985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.018036 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.018115 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:14.018946 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.019688 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.020535 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.020776 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.020862 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.023545 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:14.023550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:14.023552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:14.023581 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:14.025112 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.025578 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.025582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.025583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.025682 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.025940 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.025943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.025944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.026175 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:14.026297 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.026485 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.026610 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6423,"thread_start_us":92,"threads_started":1}
I20260923 18:11:14.024662 1612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:11:14.028456 1612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:11:14.030948 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: NON_VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: true } } }
I20260923 18:11:14.031085 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:14.032747 2883 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0: attempting to promote NON_VOTER a81c8202ca7a474bb59838f9d5902944 to VOTER
I20260923 18:11:14.033054 2883 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } }
I20260923 18:11:14.034209 2420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 LEARNER]: Refusing update from remote peer e3a6b8ab2c1340c8a837f723566263c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 18:11:14.034231 2231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 FOLLOWER]: Refusing update from remote peer e3a6b8ab2c1340c8a837f723566263c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 18:11:14.034557 2180 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 18:11:14.034744 2883 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 18:11:14.036383 2882 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a81c8202ca7a474bb59838f9d5902944 (127.27.75.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:14.036813 2231 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a81c8202ca7a474bb59838f9d5902944 (127.27.75.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:14.036859 2420 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a81c8202ca7a474bb59838f9d5902944 (127.27.75.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:14.038482 2452 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:14.038517 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:14.038601 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:14.038594 2452 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:11:14.038800 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:14.038880 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:14.039027 2882 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:14.039120 2882 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:14.040822 2891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3a6b8ab2c1340c8a837f723566263c0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:14.040956 2891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:11:14.049973 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.044547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:45679 (local address 127.0.0.1:34140)
0923 18:11:14.044802 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:14.044815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:14.044932 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:14.045770 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:14.045772 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:14.045785 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:14.045853 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.045858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.047127 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.047131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.048389 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.048398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.049525 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.049529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.049649 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.049678 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 18:11:14.049728 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 18:11:14.049813 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":58,"thread_start_us":94,"threads_started":1}
I20260923 18:11:14.050396 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.044947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.27.75.194:45679)
0923 18:11:14.045396 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.045405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.045422 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:11:14.045479 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.045482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.045624 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.045981 (+ 357us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:14.046081 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.047031 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.048501 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.049393 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.049888 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.050056 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.050116 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.050252 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":120,"threads_started":1}
I20260923 18:11:14.050869 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.044221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:44753 (local address 127.0.0.1:45992)
0923 18:11:14.045075 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:14.045086 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:14.045195 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:14.045685 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:14.045688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:14.045705 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:14.045826 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.045833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.048317 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.048320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.049235 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.049247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.050491 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.050494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.050580 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.050604 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:11:14.050673 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 18:11:14.050744 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":25,"thread_start_us":117,"threads_started":1}
I20260923 18:11:14.051227 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.044368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.27.75.193:44753)
0923 18:11:14.045186 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.045200 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.045224 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:11:14.045301 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.045307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.045535 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.045702 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:14.045933 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.046839 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.049387 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.050364 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.050642 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.050780 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.050829 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.051081 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":66,"threads_started":1}
I20260923 18:11:14.051851 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.044030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:43129 (local address 127.0.0.1:49518)
0923 18:11:14.044679 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:14.044695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:14.044788 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:14.045846 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:14.045849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:14.045866 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:14.045974 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.045981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.048299 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.048302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.050244 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.050253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.051472 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.051475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.051564 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.051585 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:11:14.051663 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 18:11:14.051731 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":84}
I20260923 18:11:14.051965 2893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.044213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.27.75.195:43129)
0923 18:11:14.045186 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.045200 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.045224 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:11:14.045301 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.045307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.045535 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.045702 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:14.046972 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.048175 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.050367 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.051154 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.051263 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.051408 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.051722 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.051828 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":141,"threads_started":1}
I20260923 18:11:14.053588 2660 tablet_service.cc:1467] Processing CreateTablet for tablet 890959cdeccd4d20b4a5081e8823f1b8 (DEFAULT_TABLE table=test-table [id=dcd6c281431a41e2bc42863575e1ad71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:14.053893 2523 tablet_service.cc:1467] Processing CreateTablet for tablet 890959cdeccd4d20b4a5081e8823f1b8 (DEFAULT_TABLE table=test-table [id=dcd6c281431a41e2bc42863575e1ad71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:14.054708 2523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890959cdeccd4d20b4a5081e8823f1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:14.054723 2660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890959cdeccd4d20b4a5081e8823f1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:14.054538 2797 tablet_service.cc:1467] Processing CreateTablet for tablet 890959cdeccd4d20b4a5081e8823f1b8 (DEFAULT_TABLE table=test-table [id=dcd6c281431a41e2bc42863575e1ad71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:14.055413 2797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890959cdeccd4d20b4a5081e8823f1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:14.062538 2897 tablet_bootstrap.cc:492] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: Bootstrap starting.
I20260923 18:11:14.063131 2899 tablet_bootstrap.cc:492] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f: Bootstrap starting.
I20260923 18:11:14.063691 2898 tablet_bootstrap.cc:492] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836: Bootstrap starting.
I20260923 18:11:14.064634 2897 tablet_bootstrap.cc:654] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:14.065286 2899 tablet_bootstrap.cc:654] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:14.065426 2897 log.cc:824] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:14.065680 2898 tablet_bootstrap.cc:654] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:14.066080 2899 log.cc:824] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:14.066407 2898 log.cc:824] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:14.067420 2897 tablet_bootstrap.cc:492] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: No bootstrap required, opened a new log
I20260923 18:11:14.067656 2897 ts_tablet_manager.cc:1397] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 18:11:14.068078 2899 tablet_bootstrap.cc:492] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f: No bootstrap required, opened a new log
I20260923 18:11:14.068291 2899 ts_tablet_manager.cc:1397] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260923 18:11:14.070521 2898 tablet_bootstrap.cc:492] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836: No bootstrap required, opened a new log
I20260923 18:11:14.070791 2898 ts_tablet_manager.cc:1397] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:11:14.071508 2897 raft_consensus.cc:348] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:14.071760 2897 raft_consensus.cc:374] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:14.071812 2897 raft_consensus.cc:729] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f45b3b090e44171bed8b23947a2bb7a, State: Initialized, Role: FOLLOWER
I20260923 18:11:14.072239 2897 consensus_queue.cc:260] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:14.072430 2899 raft_consensus.cc:348] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:14.072691 2899 raft_consensus.cc:374] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:14.072757 2899 raft_consensus.cc:729] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90362291c0604c48ae4d307f55f84b2f, State: Initialized, Role: FOLLOWER
I20260923 18:11:14.073277 2897 ts_tablet_manager.cc:1428] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:11:14.073272 2899 consensus_queue.cc:260] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:14.073470 2590 heartbeater.cc:503] Master 127.27.75.254:43469 was elected leader, sending a full tablet report...
W20260923 18:11:14.073693 2592 tablet.cc:2369] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:11:14.075285 2898 raft_consensus.cc:348] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:14.075553 2898 raft_consensus.cc:374] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:14.075600 2898 raft_consensus.cc:729] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb3fc9dd9234ff3b6e27343823ef836, State: Initialized, Role: FOLLOWER
I20260923 18:11:14.076036 2898 consensus_queue.cc:260] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:14.076870 2898 ts_tablet_manager.cc:1428] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:11:14.076871 2899 ts_tablet_manager.cc:1428] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260923 18:11:14.077551 2727 heartbeater.cc:503] Master 127.27.75.254:43469 was elected leader, sending a full tablet report...
I20260923 18:11:14.078694 2864 heartbeater.cc:503] Master 127.27.75.254:43469 was elected leader, sending a full tablet report...
I20260923 18:11:14.101774 2903 raft_consensus.cc:482] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:11:14.102017 2903 raft_consensus.cc:504] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:14.103223 2903 leader_election.cc:288] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90362291c0604c48ae4d307f55f84b2f (127.27.75.195:43129), 6cb3fc9dd9234ff3b6e27343823ef836 (127.27.75.194:45679)
I20260923 18:11:14.108384 2893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.103534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:33829 (local address 127.27.75.195:43129)
0923 18:11:14.103732 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.103736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.103817 (+ 81us) server_negotiation.cc:407] Connection header received
0923 18:11:14.103962 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.103965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.104015 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.104098 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:14.104410 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.105258 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.107099 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.107713 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.107843 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.107896 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.108203 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.108283 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260923 18:11:14.108392 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.103492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:45679 (local address 127.27.75.193:51623)
0923 18:11:14.103786 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:14.103797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:14.103873 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:14.104120 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:14.104123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:14.104141 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:14.104304 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.104311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.105400 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.105404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.106946 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.106960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.107844 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.107848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.107998 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.108047 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 18:11:14.108125 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 18:11:14.108214 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":186,"spinlock_wait_cycles":768,"thread_start_us":73,"threads_started":1}
I20260923 18:11:14.108397 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.103612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:51623 (local address 127.27.75.194:45679)
0923 18:11:14.103730 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.103735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.103859 (+ 124us) server_negotiation.cc:407] Connection header received
0923 18:11:14.103962 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.103965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.104015 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.104098 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:14.104410 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.105258 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.107099 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.107712 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.107849 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.107896 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.108188 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.108278 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 18:11:14.108927 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.103439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:43129 (local address 127.27.75.193:33829)
0923 18:11:14.103729 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:14.103744 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:14.103866 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:14.104120 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:14.104123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:14.104151 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:14.104309 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.104314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.105400 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.105404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.106946 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:14.106960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.107844 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.107848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:14.108007 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.108044 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 18:11:14.108693 (+ 649us) client_negotiation.cc:240] Negotiation successful
0923 18:11:14.108779 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":218,"thread_start_us":80,"threads_started":1}
I20260923 18:11:14.109150 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890959cdeccd4d20b4a5081e8823f1b8" candidate_uuid: "5f45b3b090e44171bed8b23947a2bb7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" is_pre_election: true
I20260923 18:11:14.109483 2680 raft_consensus.cc:2454] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f45b3b090e44171bed8b23947a2bb7a in term 0.
I20260923 18:11:14.109592 2817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890959cdeccd4d20b4a5081e8823f1b8" candidate_uuid: "5f45b3b090e44171bed8b23947a2bb7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90362291c0604c48ae4d307f55f84b2f" is_pre_election: true
I20260923 18:11:14.109860 2817 raft_consensus.cc:2454] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f45b3b090e44171bed8b23947a2bb7a in term 0.
I20260923 18:11:14.109956 2478 leader_election.cc:302] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f45b3b090e44171bed8b23947a2bb7a, 6cb3fc9dd9234ff3b6e27343823ef836; no voters:
I20260923 18:11:14.110261 2903 raft_consensus.cc:2781] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:11:14.110343 2903 raft_consensus.cc:482] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:11:14.110397 2903 raft_consensus.cc:3037] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:14.111418 2903 raft_consensus.cc:504] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:14.111775 2903 leader_election.cc:288] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [CANDIDATE]: Term 1 election: Requested vote from peers 90362291c0604c48ae4d307f55f84b2f (127.27.75.195:43129), 6cb3fc9dd9234ff3b6e27343823ef836 (127.27.75.194:45679)
I20260923 18:11:14.112162 2817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890959cdeccd4d20b4a5081e8823f1b8" candidate_uuid: "5f45b3b090e44171bed8b23947a2bb7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90362291c0604c48ae4d307f55f84b2f"
I20260923 18:11:14.112232 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "890959cdeccd4d20b4a5081e8823f1b8" candidate_uuid: "5f45b3b090e44171bed8b23947a2bb7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb3fc9dd9234ff3b6e27343823ef836"
I20260923 18:11:14.112344 2680 raft_consensus.cc:3037] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:14.112329 2817 raft_consensus.cc:3037] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:14.113719 2680 raft_consensus.cc:2454] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f45b3b090e44171bed8b23947a2bb7a in term 1.
I20260923 18:11:14.113737 2817 raft_consensus.cc:2454] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f45b3b090e44171bed8b23947a2bb7a in term 1.
I20260923 18:11:14.114035 2479 leader_election.cc:302] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f45b3b090e44171bed8b23947a2bb7a, 90362291c0604c48ae4d307f55f84b2f; no voters:
I20260923 18:11:14.114262 2903 raft_consensus.cc:2781] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:14.114434 2903 raft_consensus.cc:686] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 1 LEADER]: Becoming Leader. State: Replica: 5f45b3b090e44171bed8b23947a2bb7a, State: Running, Role: LEADER
I20260923 18:11:14.114933 2903 consensus_queue.cc:237] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:14.118924 1616 catalog_manager.cc:5495] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f45b3b090e44171bed8b23947a2bb7a (127.27.75.193). New cstate: current_term: 1 leader_uuid: "5f45b3b090e44171bed8b23947a2bb7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } health_report { overall_health: UNKNOWN } } }
I20260923 18:11:14.132329 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.129102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.27.75.193:44753)
0923 18:11:14.129252 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.129256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.129269 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:14.129377 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.129379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.129432 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.129534 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:11:14.129705 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.130198 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.131063 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.131286 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.131366 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.131586 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:11:14.131984 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:11:14.132104 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.132167 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.132217 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 18:11:14.138245 2817 raft_consensus.cc:1261] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f [term 1 FOLLOWER]: Refusing update from remote peer 5f45b3b090e44171bed8b23947a2bb7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:14.138245 2680 raft_consensus.cc:1261] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836 [term 1 FOLLOWER]: Refusing update from remote peer 5f45b3b090e44171bed8b23947a2bb7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:14.138968 2908 consensus_queue.cc:1035] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:14.139140 2903 consensus_queue.cc:1035] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:14.149318 2911 mvcc.cc:205] Tried to move back new op lower bound from 7332606255662653440 to 7332606255577722880. Current Snapshot: MvccSnapshot[applied={T|T < 7332606255662653440}]
I20260923 18:11:14.158015 1616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:14.158208 1616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:14.162812 1616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3a6b8ab2c1340c8a837f723566263c0: Sending DeleteTablet for 3 replicas of tablet 890959cdeccd4d20b4a5081e8823f1b8
I20260923 18:11:14.163542 2797 tablet_service.cc:1514] Processing DeleteTablet for tablet 890959cdeccd4d20b4a5081e8823f1b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:14 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:49518
I20260923 18:11:14.164598 2523 tablet_service.cc:1514] Processing DeleteTablet for tablet 890959cdeccd4d20b4a5081e8823f1b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:14 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:45992
I20260923 18:11:14.166144 2660 tablet_service.cc:1514] Processing DeleteTablet for tablet 890959cdeccd4d20b4a5081e8823f1b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:14 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:34140
I20260923 18:11:14.166774 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 1586
I20260923 18:11:14.168170 2922 tablet_replica.cc:330] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: stopping tablet replica
I20260923 18:11:14.168511 2922 raft_consensus.cc:2229] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:11:14.168895 2922 raft_consensus.cc:2258] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:14.171284 2922 ts_tablet_manager.cc:1905] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:14.174404 2922 ts_tablet_manager.cc:1918] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:14.174588 2922 log.cc:1196] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-0/wal/wals/890959cdeccd4d20b4a5081e8823f1b8
I20260923 18:11:14.174852 2920 tablet_replica.cc:330] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f: stopping tablet replica
I20260923 18:11:14.175204 2920 raft_consensus.cc:2229] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:14.175583 2920 raft_consensus.cc:2258] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:14.176286 2922 ts_tablet_manager.cc:1939] T 890959cdeccd4d20b4a5081e8823f1b8 P 5f45b3b090e44171bed8b23947a2bb7a: Deleting consensus metadata
I20260923 18:11:14.177239 2920 ts_tablet_manager.cc:1905] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:14.177450 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.166050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.27.75.253:33539)
0923 18:11:14.166150 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.166154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.167866 (+ 1712us) server_negotiation.cc:407] Connection header received
0923 18:11:14.167995 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.167998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.168046 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.168114 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:14.169203 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.169785 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.170815 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.171170 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.171246 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.173716 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:14.173720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:14.173723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:14.173760 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:14.175557 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.176033 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.176037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.176039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.176143 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.176437 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.176441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.176442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.176660 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:14.176779 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.177164 (+ 385us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.177304 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260923 18:11:14.177467 2476 connection.cc:441] server connection from 127.0.0.1:45992 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45992 (request call id 1) could send its response
I20260923 18:11:14.179754 2920 ts_tablet_manager.cc:1918] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:14.179842 2920 log.cc:1196] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/890959cdeccd4d20b4a5081e8823f1b8
I20260923 18:11:14.180101 2920 ts_tablet_manager.cc:1939] T 890959cdeccd4d20b4a5081e8823f1b8 P 90362291c0604c48ae4d307f55f84b2f: Deleting consensus metadata
W20260923 18:11:14.181058 2752 connection.cc:441] server connection from 127.0.0.1:49518 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49518 (request call id 1) could send its response
I20260923 18:11:14.185359 2923 tablet_replica.cc:330] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836: stopping tablet replica
I20260923 18:11:14.185740 2923 raft_consensus.cc:2229] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:14.186129 2923 raft_consensus.cc:2258] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:14.186342 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.165229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.27.75.252:40905)
0923 18:11:14.165354 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.165358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.178015 (+ 12657us) server_negotiation.cc:407] Connection header received
0923 18:11:14.178152 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.178156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.178215 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.178285 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:14.179130 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.179961 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.181105 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.181419 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.181511 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.182907 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:14.182911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:14.182914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:14.182941 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:14.184249 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.184632 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.184635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.184637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.184712 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.184951 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.184954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.184956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.185218 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:14.185336 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.185603 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.185731 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 18:11:14.187719 2923 ts_tablet_manager.cc:1905] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:14.194387 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.185961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.27.75.253:33539)
0923 18:11:14.186143 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.186147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.186278 (+ 131us) server_negotiation.cc:407] Connection header received
0923 18:11:14.186606 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.186609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.186663 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.186713 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:14.187640 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.188339 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.189201 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.189556 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.189629 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.191141 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:14.191145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:14.191148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:14.191174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:14.192680 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.193232 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.193236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.193238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.193325 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.193584 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.193588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.193590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.193811 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:14.193920 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.194131 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.194237 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260923 18:11:14.194881 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:14.186231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.27.75.252:40905)
0923 18:11:14.186337 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:14.186341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:14.187029 (+ 688us) server_negotiation.cc:407] Connection header received
0923 18:11:14.187209 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:14.187213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:14.187259 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:14.187307 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:14.188116 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.188821 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.189920 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:14.190247 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:14.190345 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:14.191848 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:14.191852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:14.191855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:14.191881 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:14.193184 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.193637 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.193640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.193642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.193716 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:14.193984 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:14.193987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:14.193989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:14.194195 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:14.194289 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:14.194618 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 18:11:14.194734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 18:11:14.197383 2923 ts_tablet_manager.cc:1918] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:14.197495 2923 log.cc:1196] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/890959cdeccd4d20b4a5081e8823f1b8
I20260923 18:11:14.197775 2923 ts_tablet_manager.cc:1939] T 890959cdeccd4d20b4a5081e8823f1b8 P 6cb3fc9dd9234ff3b6e27343823ef836: Deleting consensus metadata
W20260923 18:11:14.198868 2614 connection.cc:441] server connection from 127.0.0.1:34140 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34140 (request call id 1) could send its response
I20260923 18:11:15.153592 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.153093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.195:60137)
0923 18:11:15.153378 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.153474 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
W20260923 18:11:15.154090 2864 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:43469 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
I20260923 18:11:15.156461 2931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.156054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.193:58113)
0923 18:11:15.156309 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.156381 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":66,"threads_started":1}
W20260923 18:11:15.156730 2590 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:43469 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
I20260923 18:11:15.167590 2932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.167157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.194:36985)
0923 18:11:15.167415 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.167496 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
W20260923 18:11:15.167932 2727 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:43469 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
I20260923 18:11:15.663112 2933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3a6b8ab2c1340c8a837f723566263c0)
I20260923 18:11:15.663297 2933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } }
I20260923 18:11:15.664142 2934 raft_consensus.cc:482] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3a6b8ab2c1340c8a837f723566263c0)
I20260923 18:11:15.664232 2933 leader_election.cc:288] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3a6b8ab2c1340c8a837f723566263c0 (127.27.75.254:43469), a81c8202ca7a474bb59838f9d5902944 (127.27.75.252:40905)
I20260923 18:11:15.664336 2934 raft_consensus.cc:504] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } }
I20260923 18:11:15.665083 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.664371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39748)
0923 18:11:15.664902 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.664983 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":71,"threads_started":1}
W20260923 18:11:15.665314 2208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
I20260923 18:11:15.665330 2934 leader_election.cc:288] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3a6b8ab2c1340c8a837f723566263c0 (127.27.75.254:43469), 761778be5f0f4620ad861bb0b32a7762 (127.27.75.253:33539)
I20260923 18:11:15.665989 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.665437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39756)
0923 18:11:15.665834 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.665901 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":60,"threads_started":1}
W20260923 18:11:15.666242 2397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
I20260923 18:11:15.666522 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.665888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39762)
0923 18:11:15.666385 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.666437 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
W20260923 18:11:15.666649 2208 leader_election.cc:334] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3a6b8ab2c1340c8a837f723566263c0 (127.27.75.254:43469): Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
I20260923 18:11:15.667595 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.667365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39772)
0923 18:11:15.667470 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.667524 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 18:11:15.667680 2397 leader_election.cc:334] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3a6b8ab2c1340c8a837f723566263c0 (127.27.75.254:43469): Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
I20260923 18:11:15.668926 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.664867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34470 (local address 127.27.75.252:40905)
0923 18:11:15.665076 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:15.665081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:15.665095 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:15.665165 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:15.665168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:15.665216 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:15.665301 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:15.665716 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.666404 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.667872 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.668435 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.668513 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:15.668560 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:15.668770 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 18:11:15.668843 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260923 18:11:15.668960 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.664694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:40905 (local address 127.0.0.1:34470)
0923 18:11:15.664902 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.664916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:15.665011 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:15.665429 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:15.665432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:15.665452 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:15.665537 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:15.665543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.666518 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.666522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:15.667753 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:15.667765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.668550 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.668554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:15.668648 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:15.668676 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 18:11:15.668758 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 18:11:15.668831 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":65,"threads_started":1}
I20260923 18:11:15.669456 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "761778be5f0f4620ad861bb0b32a7762" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a81c8202ca7a474bb59838f9d5902944" is_pre_election: true
I20260923 18:11:15.669687 2420 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761778be5f0f4620ad861bb0b32a7762 in term 1.
I20260923 18:11:15.670017 2208 leader_election.cc:302] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 761778be5f0f4620ad861bb0b32a7762, a81c8202ca7a474bb59838f9d5902944; no voters: e3a6b8ab2c1340c8a837f723566263c0
I20260923 18:11:15.670246 2933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 18:11:15.670310 2933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e3a6b8ab2c1340c8a837f723566263c0)
I20260923 18:11:15.670347 2933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:11:15.670359 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.665750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:33539 (local address 127.0.0.1:36100)
0923 18:11:15.666140 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.666153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:15.666234 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:15.666936 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:15.666939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:15.666960 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:15.667055 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:15.667061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.667965 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.667968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:15.669303 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:15.669312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.670004 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.670006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:15.670070 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:15.670087 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 18:11:15.670166 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 18:11:15.670233 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":26,"thread_start_us":46,"threads_started":1}
I20260923 18:11:15.670725 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.665990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.27.75.253:33539)
0923 18:11:15.666729 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:15.666738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:15.666750 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:11:15.666790 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:15.666793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:15.666832 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:15.666898 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:15.667167 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.667873 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.669399 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.669899 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.670009 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:15.670066 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:15.670539 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 18:11:15.670613 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":98,"threads_started":1}
I20260923 18:11:15.671082 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a81c8202ca7a474bb59838f9d5902944" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "761778be5f0f4620ad861bb0b32a7762" is_pre_election: true
I20260923 18:11:15.671527 2933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } }
I20260923 18:11:15.671810 2231 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a81c8202ca7a474bb59838f9d5902944 in current term 2: Already voted for candidate 761778be5f0f4620ad861bb0b32a7762 in this term.
I20260923 18:11:15.671880 2933 leader_election.cc:288] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [CANDIDATE]: Term 2 election: Requested vote from peers e3a6b8ab2c1340c8a837f723566263c0 (127.27.75.254:43469), a81c8202ca7a474bb59838f9d5902944 (127.27.75.252:40905)
I20260923 18:11:15.672222 2397 leader_election.cc:302] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a81c8202ca7a474bb59838f9d5902944; no voters: 761778be5f0f4620ad861bb0b32a7762, e3a6b8ab2c1340c8a837f723566263c0
I20260923 18:11:15.672286 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.672049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39786)
0923 18:11:15.672139 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.672189 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 18:11:15.672430 2934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:11:15.672716 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "761778be5f0f4620ad861bb0b32a7762" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a81c8202ca7a474bb59838f9d5902944"
I20260923 18:11:15.673547 2934 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:11:15.673695 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.673444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39794)
0923 18:11:15.673538 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.673579 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 18:11:15.673835 2208 leader_election.cc:334] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e3a6b8ab2c1340c8a837f723566263c0 (127.27.75.254:43469): Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
I20260923 18:11:15.674495 2420 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761778be5f0f4620ad861bb0b32a7762 in term 2.
I20260923 18:11:15.674768 2208 leader_election.cc:302] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 761778be5f0f4620ad861bb0b32a7762, a81c8202ca7a474bb59838f9d5902944; no voters: e3a6b8ab2c1340c8a837f723566263c0
I20260923 18:11:15.674947 2933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 18:11:15.675168 2933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [term 2 LEADER]: Becoming Leader. State: Replica: 761778be5f0f4620ad861bb0b32a7762, State: Running, Role: LEADER
I20260923 18:11:15.675402 2933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } }
I20260923 18:11:15.676433 2943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 761778be5f0f4620ad861bb0b32a7762. Latest consensus state: current_term: 2 leader_uuid: "761778be5f0f4620ad861bb0b32a7762" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:15.676529 2943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:15.676784 2945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:15.678921 2945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:15.679319 2945 catalog_manager.cc:1259] Loaded cluster ID: af16d14630074fcb8a236c9641704d69
I20260923 18:11:15.679363 2945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:15.679896 2945 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:15.680275 2945 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: Loaded TSK: 0
I20260923 18:11:15.680919 2945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:11:15.724800 2221 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:11:15.728767 2221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:11:15.732738 2420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [term 2 FOLLOWER]: Refusing update from remote peer 761778be5f0f4620ad861bb0b32a7762: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260923 18:11:15.733389 2933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 18:11:15.735095 2934 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 761778be5f0f4620ad861bb0b32a7762. Latest consensus state: current_term: 2 leader_uuid: "761778be5f0f4620ad861bb0b32a7762" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:15.735242 2934 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:11:15.736159 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.732053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39800)
0923 18:11:15.735990 (+ 3937us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.736064 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3849}
I20260923 18:11:15.736966 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.736713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.0.0.1:39806)
0923 18:11:15.736844 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.736903 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 18:11:15.737169 2208 consensus_peers.cc:487] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 -> Peer e3a6b8ab2c1340c8a837f723566263c0 (127.27.75.254:43469): Couldn't send request to peer e3a6b8ab2c1340c8a837f723566263c0. Status: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 18:11:15.737905 2946 mvcc.cc:205] Tried to move back new op lower bound from 7332606262194331648 to 7332606261968805888. Current Snapshot: MvccSnapshot[applied={T|T < 7332606262194331648}]
I20260923 18:11:15.738114 2933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "761778be5f0f4620ad861bb0b32a7762" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:15.738255 2933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:15.739540 2934 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "761778be5f0f4620ad861bb0b32a7762" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a6b8ab2c1340c8a837f723566263c0" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43469 } } peers { permanent_uuid: "761778be5f0f4620ad861bb0b32a7762" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 33539 } attrs { promote: false } } peers { permanent_uuid: "a81c8202ca7a474bb59838f9d5902944" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 40905 } attrs { promote: false } } }
I20260923 18:11:15.739686 2934 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81c8202ca7a474bb59838f9d5902944 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:11:15.749265 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.745114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:44753 (local address 127.0.0.1:46002)
0923 18:11:15.745223 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.745235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:15.745314 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:15.746116 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:15.746119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:15.746135 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:15.746221 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:15.746227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.747138 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.747142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:15.747959 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:15.747969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.748831 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.748835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:15.748927 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:15.748951 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:11:15.749023 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:11:15.749099 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":16}
I20260923 18:11:15.749321 2952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.745260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.27.75.193:44753)
0923 18:11:15.745875 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:15.745880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:15.745892 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:11:15.745942 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:15.745946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:15.745992 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:15.746325 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:15.746438 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.747052 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.748082 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.748698 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.748881 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:15.748953 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:15.749036 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 18:11:15.749172 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":79,"threads_started":1}
I20260923 18:11:15.749691 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.744648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:43129 (local address 127.0.0.1:49520)
0923 18:11:15.744996 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.745009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:15.745093 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:15.745593 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:15.745596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:15.745614 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:15.745718 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:15.745725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.746721 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.746725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:15.747842 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:15.747850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.748740 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.748744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:15.749399 (+ 655us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:15.749425 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:11:15.749500 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 18:11:15.749577 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":3}
I20260923 18:11:15.749751 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.744784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.27.75.195:43129)
0923 18:11:15.745044 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:15.745058 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:15.745074 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:11:15.745438 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:15.745442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:15.745492 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:15.745575 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:15.745859 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.746606 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.747957 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.748509 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.748621 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:15.748688 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:15.749525 (+ 837us) server_negotiation.cc:299] Negotiation successful
0923 18:11:15.749621 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260923 18:11:15.750103 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.745333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:45679 (local address 127.0.0.1:34146)
0923 18:11:15.745591 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.745603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:15.746183 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:15.746450 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:15.746453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:15.746469 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:15.746557 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:15.746562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.747585 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.747589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:15.748731 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:15.748740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.749729 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.749733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:15.749828 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:15.749850 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:11:15.749916 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 18:11:15.749981 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
I20260923 18:11:15.750106 2523 tablet_service.cc:1467] Processing CreateTablet for tablet cb3e1a99c5fb455cb4cb470b2c2ca06d (DEFAULT_TABLE table=test-table [id=ef8bbc2895574b99ba3791b748518369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:15.750180 2797 tablet_service.cc:1467] Processing CreateTablet for tablet cb3e1a99c5fb455cb4cb470b2c2ca06d (DEFAULT_TABLE table=test-table [id=ef8bbc2895574b99ba3791b748518369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:15.750398 2523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb3e1a99c5fb455cb4cb470b2c2ca06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:15.750397 2797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb3e1a99c5fb455cb4cb470b2c2ca06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:15.750440 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.745803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.27.75.194:45679)
0923 18:11:15.746027 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:15.746031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:15.746044 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:15.746318 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:15.746322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:15.746367 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:15.746438 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:15.746648 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.747408 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.748861 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:15.749526 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:15.749621 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:15.749675 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:15.750243 (+ 568us) server_negotiation.cc:299] Negotiation successful
0923 18:11:15.750329 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260923 18:11:15.750896 2660 tablet_service.cc:1467] Processing CreateTablet for tablet cb3e1a99c5fb455cb4cb470b2c2ca06d (DEFAULT_TABLE table=test-table [id=ef8bbc2895574b99ba3791b748518369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:15.751111 2660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb3e1a99c5fb455cb4cb470b2c2ca06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:15.752938 2955 tablet_bootstrap.cc:492] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a: Bootstrap starting.
I20260923 18:11:15.753837 2957 tablet_bootstrap.cc:492] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836: Bootstrap starting.
I20260923 18:11:15.754016 2955 tablet_bootstrap.cc:654] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:15.754660 2957 tablet_bootstrap.cc:654] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:15.754665 2956 tablet_bootstrap.cc:492] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f: Bootstrap starting.
I20260923 18:11:15.755265 2955 tablet_bootstrap.cc:492] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a: No bootstrap required, opened a new log
I20260923 18:11:15.755362 2955 ts_tablet_manager.cc:1397] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:11:15.755450 2956 tablet_bootstrap.cc:654] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:15.755716 2957 tablet_bootstrap.cc:492] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836: No bootstrap required, opened a new log
I20260923 18:11:15.755803 2957 ts_tablet_manager.cc:1397] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 18:11:15.755810 2955 raft_consensus.cc:348] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:15.755913 2955 raft_consensus.cc:374] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:15.755945 2955 raft_consensus.cc:729] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f45b3b090e44171bed8b23947a2bb7a, State: Initialized, Role: FOLLOWER
I20260923 18:11:15.756081 2955 consensus_queue.cc:260] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:15.756201 2957 raft_consensus.cc:348] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:15.756304 2957 raft_consensus.cc:374] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:15.756337 2957 raft_consensus.cc:729] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb3fc9dd9234ff3b6e27343823ef836, State: Initialized, Role: FOLLOWER
I20260923 18:11:15.756422 2955 ts_tablet_manager.cc:1428] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:15.756451 2957 consensus_queue.cc:260] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:15.756749 2957 ts_tablet_manager.cc:1428] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:15.757633 2963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.756695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.193:60363)
0923 18:11:15.757468 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.757533 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":87,"threads_started":1}
I20260923 18:11:15.758513 2964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.757294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.194:50015)
0923 18:11:15.758368 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.758425 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":87,"threads_started":1}
I20260923 18:11:15.758826 2956 tablet_bootstrap.cc:492] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f: No bootstrap required, opened a new log
I20260923 18:11:15.758960 2956 ts_tablet_manager.cc:1397] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 18:11:15.759270 2591 heartbeater.cc:503] Master 127.27.75.253:33539 was elected leader, sending a full tablet report...
I20260923 18:11:15.759270 2963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.758057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.193:39907)
0923 18:11:15.759158 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.759208 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1034}
I20260923 18:11:15.759596 2956 raft_consensus.cc:348] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:15.759760 2956 raft_consensus.cc:374] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:15.759835 2956 raft_consensus.cc:729] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90362291c0604c48ae4d307f55f84b2f, State: Initialized, Role: FOLLOWER
I20260923 18:11:15.759969 2956 consensus_queue.cc:260] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:15.760295 2956 ts_tablet_manager.cc:1428] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:15.760519 2728 heartbeater.cc:503] Master 127.27.75.253:33539 was elected leader, sending a full tablet report...
I20260923 18:11:15.761058 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.760625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.195:55581)
0923 18:11:15.760832 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.760921 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260923 18:11:15.761981 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.761526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.195:36825)
0923 18:11:15.761853 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.761903 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":90,"threads_started":1}
I20260923 18:11:15.762355 2865 heartbeater.cc:503] Master 127.27.75.253:33539 was elected leader, sending a full tablet report...
W20260923 18:11:15.789259 2729 tablet.cc:2369] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:11:15.986503 2961 raft_consensus.cc:482] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:11:15.986670 2961 raft_consensus.cc:504] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:15.987160 2961 leader_election.cc:288] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90362291c0604c48ae4d307f55f84b2f (127.27.75.195:43129), 6cb3fc9dd9234ff3b6e27343823ef836 (127.27.75.194:45679)
I20260923 18:11:15.987612 2817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb3e1a99c5fb455cb4cb470b2c2ca06d" candidate_uuid: "5f45b3b090e44171bed8b23947a2bb7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90362291c0604c48ae4d307f55f84b2f" is_pre_election: true
I20260923 18:11:15.987702 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb3e1a99c5fb455cb4cb470b2c2ca06d" candidate_uuid: "5f45b3b090e44171bed8b23947a2bb7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" is_pre_election: true
I20260923 18:11:15.987802 2817 raft_consensus.cc:2454] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f45b3b090e44171bed8b23947a2bb7a in term 0.
I20260923 18:11:15.987839 2680 raft_consensus.cc:2454] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f45b3b090e44171bed8b23947a2bb7a in term 0.
I20260923 18:11:15.988152 2479 leader_election.cc:302] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f45b3b090e44171bed8b23947a2bb7a, 90362291c0604c48ae4d307f55f84b2f; no voters:
I20260923 18:11:15.988322 2961 raft_consensus.cc:2781] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:11:15.988379 2961 raft_consensus.cc:482] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:11:15.988426 2961 raft_consensus.cc:3037] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:15.989504 2961 raft_consensus.cc:504] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:15.989857 2961 leader_election.cc:288] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [CANDIDATE]: Term 1 election: Requested vote from peers 90362291c0604c48ae4d307f55f84b2f (127.27.75.195:43129), 6cb3fc9dd9234ff3b6e27343823ef836 (127.27.75.194:45679)
I20260923 18:11:15.990212 2817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb3e1a99c5fb455cb4cb470b2c2ca06d" candidate_uuid: "5f45b3b090e44171bed8b23947a2bb7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90362291c0604c48ae4d307f55f84b2f"
I20260923 18:11:15.990345 2817 raft_consensus.cc:3037] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:15.990353 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb3e1a99c5fb455cb4cb470b2c2ca06d" candidate_uuid: "5f45b3b090e44171bed8b23947a2bb7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb3fc9dd9234ff3b6e27343823ef836"
I20260923 18:11:15.990490 2680 raft_consensus.cc:3037] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:15.991322 2817 raft_consensus.cc:2454] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f45b3b090e44171bed8b23947a2bb7a in term 1.
I20260923 18:11:15.991598 2479 leader_election.cc:302] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f45b3b090e44171bed8b23947a2bb7a, 90362291c0604c48ae4d307f55f84b2f; no voters:
I20260923 18:11:15.991613 2680 raft_consensus.cc:2454] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f45b3b090e44171bed8b23947a2bb7a in term 1.
I20260923 18:11:15.991820 2961 raft_consensus.cc:2781] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:15.992039 2961 raft_consensus.cc:686] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 1 LEADER]: Becoming Leader. State: Replica: 5f45b3b090e44171bed8b23947a2bb7a, State: Running, Role: LEADER
I20260923 18:11:15.992339 2961 consensus_queue.cc:237] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } }
I20260923 18:11:15.992758 2963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:15.992526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.193:55207)
0923 18:11:15.992632 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:15.992692 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 18:11:15.994655 2221 catalog_manager.cc:5495] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f45b3b090e44171bed8b23947a2bb7a (127.27.75.193). New cstate: current_term: 1 leader_uuid: "5f45b3b090e44171bed8b23947a2bb7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f45b3b090e44171bed8b23947a2bb7a" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 } health_report { overall_health: UNKNOWN } } }
W20260923 18:11:15.995108 2866 tablet.cc:2369] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:11:16.049458 2952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:16.046209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.27.75.193:44753)
0923 18:11:16.046329 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:16.046333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:16.046346 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:16.046553 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:16.046556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:16.046609 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:16.046677 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:11:16.046919 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:16.047494 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:16.048376 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:16.048628 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:16.048712 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:16.048955 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:11:16.049034 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:11:16.049104 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:16.049269 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 18:11:16.049335 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 18:11:16.051595 2817 raft_consensus.cc:1261] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f [term 1 FOLLOWER]: Refusing update from remote peer 5f45b3b090e44171bed8b23947a2bb7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:16.051587 2680 raft_consensus.cc:1261] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836 [term 1 FOLLOWER]: Refusing update from remote peer 5f45b3b090e44171bed8b23947a2bb7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:16.052073 2961 consensus_queue.cc:1035] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90362291c0604c48ae4d307f55f84b2f" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:16.052229 2968 consensus_queue.cc:1035] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cb3fc9dd9234ff3b6e27343823ef836" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:16.053640 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:16.053341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.195:52313)
0923 18:11:16.053500 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:16.053561 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 18:11:16.053946 2964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:16.053657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.194:58127)
0923 18:11:16.053821 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:16.053880 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 18:11:16.057621 2973 mvcc.cc:205] Tried to move back new op lower bound from 7332606263503048704 to 7332606263267770368. Current Snapshot: MvccSnapshot[applied={T|T < 7332606263503048704}]
I20260923 18:11:16.057796 2974 mvcc.cc:205] Tried to move back new op lower bound from 7332606263503048704 to 7332606263267770368. Current Snapshot: MvccSnapshot[applied={T|T < 7332606263503048704}]
I20260923 18:11:16.058758 2964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:16.058146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.194:37463)
0923 18:11:16.058635 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:16.058689 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418}
I20260923 18:11:16.059755 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:16.057438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.195:55585)
0923 18:11:16.057721 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:16.057778 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260923 18:11:16.060377 2963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:16.060140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.193:42191)
0923 18:11:16.060252 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:16.060305 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 18:11:16.061431 2963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:16.061175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43469 (local address 127.27.75.193:33699)
0923 18:11:16.061316 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:16.061366 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 18:11:16.062556 2221 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:16.062696 2221 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:16.066269 2221 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 761778be5f0f4620ad861bb0b32a7762: Sending DeleteTablet for 3 replicas of tablet cb3e1a99c5fb455cb4cb470b2c2ca06d
I20260923 18:11:16.066859 2797 tablet_service.cc:1514] Processing DeleteTablet for tablet cb3e1a99c5fb455cb4cb470b2c2ca06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:16 UTC) from {username='oryx', principal='oryx/127.27.75.253@KRBTEST.COM'} at 127.0.0.1:49520
I20260923 18:11:16.067040 2523 tablet_service.cc:1514] Processing DeleteTablet for tablet cb3e1a99c5fb455cb4cb470b2c2ca06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:16 UTC) from {username='oryx', principal='oryx/127.27.75.253@KRBTEST.COM'} at 127.0.0.1:46002
I20260923 18:11:16.067221 2660 tablet_service.cc:1514] Processing DeleteTablet for tablet cb3e1a99c5fb455cb4cb470b2c2ca06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:16 UTC) from {username='oryx', principal='oryx/127.27.75.253@KRBTEST.COM'} at 127.0.0.1:34146
I20260923 18:11:16.067413 2982 tablet_replica.cc:330] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a: stopping tablet replica
I20260923 18:11:16.067566 2981 tablet_replica.cc:330] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f: stopping tablet replica
I20260923 18:11:16.067595 2983 tablet_replica.cc:330] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836: stopping tablet replica
I20260923 18:11:16.067579 2982 raft_consensus.cc:2229] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:11:16.067699 2981 raft_consensus.cc:2229] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:16.067742 2983 raft_consensus.cc:2229] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:16.067834 2981 raft_consensus.cc:2258] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:16.067874 2983 raft_consensus.cc:2258] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:16.067890 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 2365
I20260923 18:11:16.068113 2982 raft_consensus.cc:2258] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 5f45b3b090e44171bed8b23947a2bb7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:16.068771 2983 ts_tablet_manager.cc:1905] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:16.070802 2981 ts_tablet_manager.cc:1905] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:16.071053 2983 ts_tablet_manager.cc:1918] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:16.071122 2983 log.cc:1196] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/cb3e1a99c5fb455cb4cb470b2c2ca06d
I20260923 18:11:16.071359 2983 ts_tablet_manager.cc:1939] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 6cb3fc9dd9234ff3b6e27343823ef836: Deleting consensus metadata
I20260923 18:11:16.073318 2981 ts_tablet_manager.cc:1918] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:16.073325 2207 catalog_manager.cc:4841] TS 6cb3fc9dd9234ff3b6e27343823ef836 (127.27.75.194:45679): tablet cb3e1a99c5fb455cb4cb470b2c2ca06d (table test-table [id=ef8bbc2895574b99ba3791b748518369]) successfully deleted
I20260923 18:11:16.073400 2981 log.cc:1196] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/cb3e1a99c5fb455cb4cb470b2c2ca06d
I20260923 18:11:16.073657 2981 ts_tablet_manager.cc:1939] T cb3e1a99c5fb455cb4cb470b2c2ca06d P 90362291c0604c48ae4d307f55f84b2f: Deleting consensus metadata
I20260923 18:11:16.074386 2208 catalog_manager.cc:4841] TS 90362291c0604c48ae4d307f55f84b2f (127.27.75.195:43129): tablet cb3e1a99c5fb455cb4cb470b2c2ca06d (table test-table [id=ef8bbc2895574b99ba3791b748518369]) successfully deleted
I20260923 18:11:16.076308 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 2597
I20260923 18:11:16.076468 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:16.076197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:44753 (local address 127.0.0.1:46018)
0923 18:11:16.076329 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:16.076381 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 18:11:16.077152 2207 catalog_manager.cc:4584] TS 5f45b3b090e44171bed8b23947a2bb7a (127.27.75.193:44753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb3e1a99c5fb455cb4cb470b2c2ca06d: Network error: Client connection negotiation failed: client connection to 127.27.75.193:44753: connect: Connection refused (error 111)
I20260923 18:11:16.081846 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 2733
I20260923 18:11:16.087213 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 2082
I20260923 18:11:16.092291 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 2186
2026-09-23T18:11:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5352 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:11:16 dist-test-slave-wsx5 krb5kdc[2986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:11:16 dist-test-slave-wsx5 krb5kdc[2986](info): set up 2 sockets
Sep 23 18:11:16 dist-test-slave-wsx5 krb5kdc[2986](info): commencing operation
krb5kdc: starting...
W20260923 18:11:17.059652 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:11:17 dist-test-slave-wsx5 krb5kdc[2986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.254.keytab.
Entry for principal oryx/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.254.keytab.
I20260923 18:11:17.174347 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:40937
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.254.keytab
--principal=oryx/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:40937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:17.282409 2999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:17.282703 2999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:17.282759 2999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:17.286963 2999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:17.287042 2999 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 18:11:17.287065 2999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:17.287089 2999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:17.287109 2999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:17.287130 2999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:17.292722 2999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:40937
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.254.keytab
--principal=oryx/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:40937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:17.293874 2999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:17.294931 2999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:17.303757 3004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:17.303807 3005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:17.303890 2999 server_base.cc:1034] running on GCE node
W20260923 18:11:17.303771 3007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:17.304709 2999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:17.305958 2999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:17.307145 2999 hybrid_clock.cc:648] HybridClock initialized: now 1790187077307125 us; error 40 us; skew 500 ppm
Sep 23 18:11:17 dist-test-slave-wsx5 krb5kdc[2986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187077, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:17.310055 2999 init.cc:377] Logged in from keytab as oryx/127.27.75.254@KRBTEST.COM (short username oryx)
I20260923 18:11:17.310990 2999 webserver.cc:460] Webserver started at http://127.27.75.254:35343/ using document root <none> and password file <none>
I20260923 18:11:17.311525 2999 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:17.311590 2999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:17.311802 2999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:17.313901 2999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "edd08fb4e557423492e25a12388421fa"
format_stamp: "Formatted at 2026-09-23 18:11:17 on dist-test-slave-wsx5"
server_key: "5f39c00dcb68d36185fd23f38a3ea6e2"
server_key_iv: "f1bef190066ab83824801341a5605958"
server_key_version: "encryptionkey@0"
I20260923 18:11:17.314468 2999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "edd08fb4e557423492e25a12388421fa"
format_stamp: "Formatted at 2026-09-23 18:11:17 on dist-test-slave-wsx5"
server_key: "5f39c00dcb68d36185fd23f38a3ea6e2"
server_key_iv: "f1bef190066ab83824801341a5605958"
server_key_version: "encryptionkey@0"
I20260923 18:11:17.318310 2999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:11:17.320828 3014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:17.322178 2999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:11:17.322299 2999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "edd08fb4e557423492e25a12388421fa"
format_stamp: "Formatted at 2026-09-23 18:11:17 on dist-test-slave-wsx5"
server_key: "5f39c00dcb68d36185fd23f38a3ea6e2"
server_key_iv: "f1bef190066ab83824801341a5605958"
server_key_version: "encryptionkey@0"
I20260923 18:11:17.322403 2999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:17.350673 2999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:17.353554 2999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:17.353744 2999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:17.361200 2999 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:40937
I20260923 18:11:17.361228 3077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:40937 every 8 connection(s)
I20260923 18:11:17.362149 2999 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:17.364921 3078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:17.370563 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 2999
I20260923 18:11:17.370698 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:17.370693 3078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa: Bootstrap starting.
I20260923 18:11:17.370965 27951 external_mini_cluster.cc:1428] Setting key 7513ea27e142f94bafd709d9a0148cc8
I20260923 18:11:17.373257 3078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:17.374014 3078 log.cc:824] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:17.375988 3078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa: No bootstrap required, opened a new log
Sep 23 18:11:17 dist-test-slave-wsx5 krb5kdc[2986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
I20260923 18:11:17.379745 3078 raft_consensus.cc:348] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd08fb4e557423492e25a12388421fa" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40937 } }
I20260923 18:11:17.379999 3078 raft_consensus.cc:374] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:17.380036 3078 raft_consensus.cc:729] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd08fb4e557423492e25a12388421fa, State: Initialized, Role: FOLLOWER
I20260923 18:11:17.380484 3078 consensus_queue.cc:260] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd08fb4e557423492e25a12388421fa" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40937 } }
I20260923 18:11:17.380653 3078 raft_consensus.cc:388] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:17.380724 3078 raft_consensus.cc:482] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:17.380829 3078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:17.381793 3078 raft_consensus.cc:504] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd08fb4e557423492e25a12388421fa" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40937 } }
I20260923 18:11:17.382117 3078 leader_election.cc:302] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edd08fb4e557423492e25a12388421fa; no voters:
I20260923 18:11:17.382411 3078 leader_election.cc:288] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:17.382532 3083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:17.382728 3083 raft_consensus.cc:686] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [term 1 LEADER]: Becoming Leader. State: Replica: edd08fb4e557423492e25a12388421fa, State: Running, Role: LEADER
I20260923 18:11:17.383015 3083 consensus_queue.cc:237] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd08fb4e557423492e25a12388421fa" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40937 } }
I20260923 18:11:17.383713 3078 sys_catalog.cc:564] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:17.384810 3084 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edd08fb4e557423492e25a12388421fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd08fb4e557423492e25a12388421fa" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40937 } } }
I20260923 18:11:17.384929 3084 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:17.385360 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:17.372287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.27.75.254:40937)
0923 18:11:17.372712 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:17.372728 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:17.372765 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:11:17.373510 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:17.373530 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:17.373879 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:17.374192 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:17.374953 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.376010 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.376855 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.377391 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.377537 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:17.380184 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:17.380189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:17.380203 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:17.380234 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:17.382054 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:17.382625 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:17.382632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:17.382639 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:17.382791 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:17.383108 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:17.383111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:17.383113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:17.383530 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:17.383702 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:17.384023 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 18:11:17.384330 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
I20260923 18:11:17.385360 3093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:17.385933 3085 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader edd08fb4e557423492e25a12388421fa. Latest consensus state: current_term: 1 leader_uuid: "edd08fb4e557423492e25a12388421fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd08fb4e557423492e25a12388421fa" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 40937 } } }
I20260923 18:11:17.386076 3085 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:17.389070 3093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:17.395008 3093 catalog_manager.cc:1347] Generated new cluster ID: 36bacbdd4a9b4485b2b82e2a5fb5f688
I20260923 18:11:17.395089 3093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:17.405530 3093 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:17.406483 3093 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:17.413404 3093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa: Generated new TSK 0
I20260923 18:11:17.414176 3093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab.
Entry for principal oryx/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab.
I20260923 18:11:17.450302 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:40937
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab
--principal=oryx/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:17.567056 3104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:17.567360 3104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:17.567405 3104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:17.571709 3104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:17.571784 3104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:17.571825 3104 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:11:17.571858 3104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:17.577461 3104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab
--principal=oryx/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.75.254:40937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:17.578577 3104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:17.579593 3104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:17.589808 3112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:17.589808 3110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:17.589851 3109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:17.590188 3104 server_base.cc:1034] running on GCE node
I20260923 18:11:17.590574 3104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:17.591233 3104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:17.592396 3104 hybrid_clock.cc:648] HybridClock initialized: now 1790187077592364 us; error 43 us; skew 500 ppm
Sep 23 18:11:17 dist-test-slave-wsx5 krb5kdc[2986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187077, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:17.595212 3104 init.cc:377] Logged in from keytab as oryx/127.27.75.193@KRBTEST.COM (short username oryx)
I20260923 18:11:17.596073 3104 webserver.cc:460] Webserver started at http://127.27.75.193:37063/ using document root <none> and password file <none>
I20260923 18:11:17.596601 3104 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:17.596660 3104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:17.596833 3104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:17.598898 3104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "c23e58d2dfff400581b159a7d171afdf"
format_stamp: "Formatted at 2026-09-23 18:11:17 on dist-test-slave-wsx5"
server_key: "1367346cfe56443b52a3041c92053017"
server_key_iv: "7f9f19e3fd3c45a1e8a8cbad4f203dd5"
server_key_version: "encryptionkey@0"
I20260923 18:11:17.599442 3104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "c23e58d2dfff400581b159a7d171afdf"
format_stamp: "Formatted at 2026-09-23 18:11:17 on dist-test-slave-wsx5"
server_key: "1367346cfe56443b52a3041c92053017"
server_key_iv: "7f9f19e3fd3c45a1e8a8cbad4f203dd5"
server_key_version: "encryptionkey@0"
I20260923 18:11:17.603459 3104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:11:17.606254 3119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:17.607570 3104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:17.607692 3104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "c23e58d2dfff400581b159a7d171afdf"
format_stamp: "Formatted at 2026-09-23 18:11:17 on dist-test-slave-wsx5"
server_key: "1367346cfe56443b52a3041c92053017"
server_key_iv: "7f9f19e3fd3c45a1e8a8cbad4f203dd5"
server_key_version: "encryptionkey@0"
I20260923 18:11:17.607801 3104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:17.620559 3104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:17.623643 3104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:17.623823 3104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:17.625317 3104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:17.625370 3104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:17.625425 3104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:17.625451 3104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:11:17 dist-test-slave-wsx5 krb5kdc[2986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187077, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
I20260923 18:11:17.635519 3104 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:42551
I20260923 18:11:17.635696 3234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:42551 every 8 connection(s)
I20260923 18:11:17.636528 3104 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:17.639161 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:17.626078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:37769 (local address 127.27.75.254:40937)
0923 18:11:17.626239 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:17.626244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:17.627161 (+ 917us) server_negotiation.cc:407] Connection header received
0923 18:11:17.628136 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:17.628140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:17.628203 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:17.628303 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:17.629893 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.630754 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.632220 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.632459 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.632563 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:17.635833 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:17.635838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:17.635840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:17.635871 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:17.637375 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:17.637894 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:17.637897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:17.637898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:17.637978 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:17.638241 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:17.638243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:17.638244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:17.638420 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:17.638491 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:17.638940 (+ 449us) server_negotiation.cc:299] Negotiation successful
0923 18:11:17.639038 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:11:17.639928 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:17.626493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:40937 (local address 127.27.75.193:37769)
0923 18:11:17.626979 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:17.627051 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:17.627906 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:17.628562 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:17.628571 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:17.629004 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:17.629730 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:17.629751 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.630899 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.630902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:17.632088 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:17.632100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.632617 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.632623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:17.632846 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:17.633788 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:17.633811 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:17.635581 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:17.637481 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:17.637486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:17.637497 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:17.637804 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:17.638046 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:17.638048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:17.638049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:17.638177 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:17.638489 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:17.638493 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:17.638761 (+ 268us) client_negotiation.cc:769] Sending connection context
0923 18:11:17.638911 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 18:11:17.639153 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":116,"threads_started":1}
I20260923 18:11:17.641242 3235 heartbeater.cc:347] Connected to a master server at 127.27.75.254:40937
I20260923 18:11:17.641384 3235 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:17.641886 3235 heartbeater.cc:511] Master 127.27.75.254:40937 requested a full tablet report, sending...
I20260923 18:11:17.643440 3030 ts_manager.cc:194] Registered new tserver with Master: c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551)
I20260923 18:11:17.644716 3030 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:37769
I20260923 18:11:17.646711 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 3104
I20260923 18:11:17.646826 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:11:17.647114 27951 external_mini_cluster.cc:1428] Setting key 394d1e46d47c6e1178892e36b82f1a3d
I20260923 18:11:17.650054 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:17.643134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:40937 (local address 127.27.75.193:50209)
0923 18:11:17.643424 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:17.643439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:17.643539 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:17.643773 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:17.643776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:17.643978 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:17.644100 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:17.644107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.644820 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.644824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:17.645567 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:17.645576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.645997 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.646000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:17.646091 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:17.646613 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:17.646627 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:17.647042 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:17.648696 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:17.648699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:17.648701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:17.648945 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:17.649229 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:17.649231 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:17.649232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:17.649305 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:17.649653 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:17.649655 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:17.649725 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 18:11:17.649798 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 18:11:17.649903 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
I20260923 18:11:17.650099 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:17.643248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:50209 (local address 127.27.75.254:40937)
0923 18:11:17.643365 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:17.643369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:17.643506 (+ 137us) server_negotiation.cc:407] Connection header received
0923 18:11:17.643621 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:17.643624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:17.643671 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:17.643739 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:17.644205 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.644697 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.645671 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.645892 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.645971 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:17.647183 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:17.647187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:17.647190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:17.647218 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:17.648596 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:17.649029 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:17.649032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:17.649034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:17.649151 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:17.649382 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:17.649385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:17.649387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:17.649577 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:17.649669 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:17.649856 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 18:11:17.649963 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for oryx/127.27.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab.
Entry for principal oryx/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab.
I20260923 18:11:17.674316 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:40937
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab
--principal=oryx/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:17.787324 3244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:17.787621 3244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:17.787669 3244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:17.791917 3244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:17.791996 3244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:17.792038 3244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:11:17.792071 3244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:17.797521 3244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab
--principal=oryx/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.75.254:40937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:17.798602 3244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:17.799573 3244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:17.809229 3250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:17.809232 3249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:17.809232 3252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:17.809439 3244 server_base.cc:1034] running on GCE node
I20260923 18:11:17.809978 3244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:17.810643 3244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:17.811862 3244 hybrid_clock.cc:648] HybridClock initialized: now 1790187077811829 us; error 49 us; skew 500 ppm
Sep 23 18:11:17 dist-test-slave-wsx5 krb5kdc[2986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187077, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:17.814796 3244 init.cc:377] Logged in from keytab as oryx/127.27.75.194@KRBTEST.COM (short username oryx)
I20260923 18:11:17.815702 3244 webserver.cc:460] Webserver started at http://127.27.75.194:37587/ using document root <none> and password file <none>
I20260923 18:11:17.816252 3244 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:17.816319 3244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:17.816517 3244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:17.818627 3244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "3e8d0cb1a38f40d89624f5642fad9ebe"
format_stamp: "Formatted at 2026-09-23 18:11:17 on dist-test-slave-wsx5"
server_key: "cdbd386c7c0c59f966053ade39743aa3"
server_key_iv: "e017ec1b241d53d59bd9a1c9818447d8"
server_key_version: "encryptionkey@0"
I20260923 18:11:17.819181 3244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "3e8d0cb1a38f40d89624f5642fad9ebe"
format_stamp: "Formatted at 2026-09-23 18:11:17 on dist-test-slave-wsx5"
server_key: "cdbd386c7c0c59f966053ade39743aa3"
server_key_iv: "e017ec1b241d53d59bd9a1c9818447d8"
server_key_version: "encryptionkey@0"
I20260923 18:11:17.823102 3244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:11:17.825796 3259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:17.827304 3244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:17.827419 3244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "3e8d0cb1a38f40d89624f5642fad9ebe"
format_stamp: "Formatted at 2026-09-23 18:11:17 on dist-test-slave-wsx5"
server_key: "cdbd386c7c0c59f966053ade39743aa3"
server_key_iv: "e017ec1b241d53d59bd9a1c9818447d8"
server_key_version: "encryptionkey@0"
I20260923 18:11:17.827529 3244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:17.841660 3244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:17.845314 3244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:17.845515 3244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:17.846974 3244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:17.847025 3244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:17.847077 3244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:17.847102 3244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:11:17 dist-test-slave-wsx5 krb5kdc[2986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187077, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
I20260923 18:11:17.857596 3244 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:40785
I20260923 18:11:17.857622 3374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:40785 every 8 connection(s)
I20260923 18:11:17.858618 3244 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:17.860838 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 3244
I20260923 18:11:17.860980 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:11:17.861331 27951 external_mini_cluster.cc:1428] Setting key e7971246562673d34c2f10f4135e1089
I20260923 18:11:17.861770 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:17.847921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:42797 (local address 127.27.75.254:40937)
0923 18:11:17.848080 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:17.848085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:17.848954 (+ 869us) server_negotiation.cc:407] Connection header received
0923 18:11:17.850169 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:17.850173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:17.850242 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:17.850350 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:17.851803 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.852563 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.854294 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.854625 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.854752 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:17.857841 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:17.857846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:17.857848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:17.857879 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:17.859425 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:17.860082 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:17.860085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:17.860086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:17.860187 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:17.860509 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:17.860513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:17.860514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:17.860783 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:17.860895 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:17.861474 (+ 579us) server_negotiation.cc:299] Negotiation successful
0923 18:11:17.861612 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:11:17.862658 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:17.848300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:40937 (local address 127.27.75.194:42797)
0923 18:11:17.848766 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:17.848819 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:17.849938 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:17.850653 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:17.850661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:17.851088 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:17.851632 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:17.851649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.852724 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.852727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:17.854111 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:17.854135 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.854820 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.854824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:17.854994 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:17.855696 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:17.855721 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:17.857632 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:17.859629 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:17.859635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:17.859649 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:17.859932 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:17.860265 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:17.860267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:17.860269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:17.860394 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:17.860881 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:17.860886 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:17.861217 (+ 331us) client_negotiation.cc:769] Sending connection context
0923 18:11:17.861377 (+ 160us) client_negotiation.cc:240] Negotiation successful
0923 18:11:17.861709 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260923 18:11:17.863919 3375 heartbeater.cc:347] Connected to a master server at 127.27.75.254:40937
I20260923 18:11:17.864073 3375 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:17.864671 3375 heartbeater.cc:511] Master 127.27.75.254:40937 requested a full tablet report, sending...
I20260923 18:11:17.865669 3031 ts_manager.cc:194] Registered new tserver with Master: 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785)
I20260923 18:11:17.866300 3031 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:42797
WARNING: no policy specified for oryx/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:11:17.873147 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:17.866169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:40937 (local address 127.27.75.194:51569)
0923 18:11:17.866502 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:17.866517 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:17.866616 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:17.866878 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:17.866880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:17.867087 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:17.867221 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:17.867229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.867946 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.867950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:17.868518 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:17.868525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.868949 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.868952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:17.869039 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:17.869655 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:17.869666 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:17.869986 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:17.871650 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:17.871653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:17.871655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:17.871847 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:17.872097 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:17.872099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:17.872101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:17.872175 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:17.872602 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:17.872605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:17.872697 (+ 92us) client_negotiation.cc:769] Sending connection context
0923 18:11:17.872792 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 18:11:17.872933 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260923 18:11:17.873214 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:17.866598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:51569 (local address 127.27.75.254:40937)
0923 18:11:17.866706 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:17.866710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:17.866721 (+ 11us) server_negotiation.cc:407] Connection header received
0923 18:11:17.866753 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:17.866755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:17.866785 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:17.866846 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:17.867324 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.867834 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.868629 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:17.868852 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:17.868938 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:17.870128 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:17.870131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:17.870134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:17.870159 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:17.871541 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:17.871936 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:17.871939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:17.871941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:17.872022 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:17.872257 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:17.872259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:17.872261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:17.872488 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:17.872568 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:17.872904 (+ 336us) server_negotiation.cc:299] Negotiation successful
0923 18:11:17.873024 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab.
Entry for principal oryx/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab.
I20260923 18:11:17.888620 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:40937
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab
--principal=oryx/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:18.000487 3384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:18.000753 3384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:18.000813 3384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:18.004771 3384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:18.004832 3384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:18.004886 3384 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 18:11:18.004942 3384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:18.010221 3384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab
--principal=oryx/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.75.254:40937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:18.011296 3384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:18.012312 3384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:18.021960 3392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:18.021960 3389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:18.021965 3390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:18.022080 3384 server_base.cc:1034] running on GCE node
I20260923 18:11:18.022636 3384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:18.023285 3384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:18.024446 3384 hybrid_clock.cc:648] HybridClock initialized: now 1790187078024423 us; error 35 us; skew 500 ppm
Sep 23 18:11:18 dist-test-slave-wsx5 krb5kdc[2986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187078, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:18.027212 3384 init.cc:377] Logged in from keytab as oryx/127.27.75.195@KRBTEST.COM (short username oryx)
I20260923 18:11:18.028079 3384 webserver.cc:460] Webserver started at http://127.27.75.195:38851/ using document root <none> and password file <none>
I20260923 18:11:18.028617 3384 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:18.028687 3384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:18.028878 3384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:18.030984 3384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "68aa72c49fbe4414912cf85ae3df635b"
format_stamp: "Formatted at 2026-09-23 18:11:18 on dist-test-slave-wsx5"
server_key: "482b96ea49a284019eb0f7f4a87b1f58"
server_key_iv: "74e4c3a99e692a23c31c7952d1110f34"
server_key_version: "encryptionkey@0"
I20260923 18:11:18.031518 3384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "68aa72c49fbe4414912cf85ae3df635b"
format_stamp: "Formatted at 2026-09-23 18:11:18 on dist-test-slave-wsx5"
server_key: "482b96ea49a284019eb0f7f4a87b1f58"
server_key_iv: "74e4c3a99e692a23c31c7952d1110f34"
server_key_version: "encryptionkey@0"
I20260923 18:11:18.035213 3384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 18:11:18.037752 3399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:18.039134 3384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:18.039283 3384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "68aa72c49fbe4414912cf85ae3df635b"
format_stamp: "Formatted at 2026-09-23 18:11:18 on dist-test-slave-wsx5"
server_key: "482b96ea49a284019eb0f7f4a87b1f58"
server_key_iv: "74e4c3a99e692a23c31c7952d1110f34"
server_key_version: "encryptionkey@0"
I20260923 18:11:18.039390 3384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:18.051621 3384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:18.054878 3384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:18.055076 3384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:18.056576 3384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:18.056628 3384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:18.056694 3384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:18.056720 3384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 18:11:18 dist-test-slave-wsx5 krb5kdc[2986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187078, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
I20260923 18:11:18.066946 3384 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:46357
I20260923 18:11:18.067076 3514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:46357 every 8 connection(s)
I20260923 18:11:18.067976 3384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:18.070353 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.057383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:40991 (local address 127.27.75.254:40937)
0923 18:11:18.057540 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.057544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.058441 (+ 897us) server_negotiation.cc:407] Connection header received
0923 18:11:18.059546 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.059551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.059613 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.059673 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:18.061236 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.061828 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.063069 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.063362 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.063465 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.066784 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:18.066789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:18.066792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:18.066823 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:18.068262 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:18.068876 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:18.068880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:18.068882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:18.068966 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:18.069370 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:18.069373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:18.069374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:18.069580 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:18.069681 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.070127 (+ 446us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.070231 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:11:18.071108 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.057797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:40937 (local address 127.27.75.195:40991)
0923 18:11:18.058251 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.058305 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.059276 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.060108 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.060117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.060510 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:18.061074 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.061090 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.061957 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.061961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.062938 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.062948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.063610 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.063650 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.063892 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.064612 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:18.064628 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:18.066506 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:18.068403 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:18.068409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:18.068424 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:18.068752 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:18.069067 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:18.069070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:18.069071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:18.069251 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:18.069689 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:18.069693 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:18.069919 (+ 226us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.070090 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.070341 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":123,"threads_started":1}
I20260923 18:11:18.072216 3515 heartbeater.cc:347] Connected to a master server at 127.27.75.254:40937
I20260923 18:11:18.072360 3515 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:18.072901 3515 heartbeater.cc:511] Master 127.27.75.254:40937 requested a full tablet report, sending...
I20260923 18:11:18.074191 3030 ts_manager.cc:194] Registered new tserver with Master: 68aa72c49fbe4414912cf85ae3df635b (127.27.75.195:46357)
I20260923 18:11:18.075006 3030 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:40991
I20260923 18:11:18.075244 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 3384
I20260923 18:11:18.075373 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:11:18.075704 27951 external_mini_cluster.cc:1428] Setting key 6201bcc06388ae2bb49addde82513572
I20260923 18:11:18.077819 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:11:18.082752 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.074546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:50363 (local address 127.27.75.254:40937)
0923 18:11:18.074645 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.074649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.074914 (+ 265us) server_negotiation.cc:407] Connection header received
0923 18:11:18.075032 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.075034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.075080 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.075163 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:18.075817 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.076412 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.077456 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.077770 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.077861 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.079208 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:18.079211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:18.079213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:18.079237 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:18.080880 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:18.081455 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:18.081458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:18.081463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:18.081552 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:18.081861 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:18.081864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:18.081865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:18.082093 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:18.082184 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.082470 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.082618 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 18:11:18.082799 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.074494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:40937 (local address 127.27.75.195:50363)
0923 18:11:18.074837 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.074852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.074947 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.075277 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.075280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.075494 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:18.075686 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.075694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.076573 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.076576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.077350 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.077359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.077908 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.077912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.078020 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.078490 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:18.078505 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:18.079000 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:18.081023 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:18.081027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:18.081028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:18.081326 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:18.081647 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:18.081650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:18.081652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:18.081747 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:18.082189 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:18.082192 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:18.082302 (+ 110us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.082404 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.082586 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
Sep 23 18:11:18 dist-test-slave-wsx5 krb5kdc[2986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187078, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 18:11:18 dist-test-slave-wsx5 krb5kdc[2986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:18 dist-test-slave-wsx5 krb5kdc[2986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:11:18.102903 3081 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:18.096507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.27.75.254:40937)
0923 18:11:18.096644 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.096647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.096668 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:11:18.096710 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.096713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.096753 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.096823 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:18.097675 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.098224 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.099107 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.099332 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.099424 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.102742 (+ 3318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60488: BlockingRecv error: recv got EOF from 127.0.0.1:60488 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 18:11:18 dist-test-slave-wsx5 krb5kdc[2986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187078, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
I20260923 18:11:18.112980 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.104557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.27.75.254:40937)
0923 18:11:18.104697 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.104701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.104813 (+ 112us) server_negotiation.cc:407] Connection header received
0923 18:11:18.104947 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.104950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.104993 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.105059 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:18.105719 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.106239 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.107057 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.107270 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.107358 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.109792 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:18.109796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:18.109798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:18.109828 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:18.111304 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:18.111822 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:18.111825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:18.111827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:18.111922 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:18.112207 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:18.112210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:18.112211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:18.112404 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:18.112511 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.112744 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.112850 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:11:18.116182 3030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:11:18.119298 3030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:11:18.134498 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.129680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.27.75.194:40785)
0923 18:11:18.130022 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.130035 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.130057 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:11:18.130129 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.130134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.130354 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.130496 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:18.130805 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.131634 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.133020 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.133807 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.133917 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.134088 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.134239 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.134343 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":126,"threads_started":1}
I20260923 18:11:18.134490 3541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.129298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:40785 (local address 127.0.0.1:59320)
0923 18:11:18.129747 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.129758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.129828 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.130563 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.130566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.130580 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:18.130684 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.130692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.131745 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.131749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.132893 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.132902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.133925 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.133929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.134039 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.134081 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.134198 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.134326 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":19,"thread_start_us":72,"threads_started":1}
I20260923 18:11:18.135248 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.128556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:42551 (local address 127.0.0.1:34282)
0923 18:11:18.129415 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.129472 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.129614 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.130442 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.130449 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.130471 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:18.130690 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.130697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.132212 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.132215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.133632 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.133643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.134856 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.134859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.134950 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.134972 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.135043 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.135115 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":136,"thread_start_us":83,"threads_started":1}
I20260923 18:11:18.135430 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.128884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.27.75.195:46357)
0923 18:11:18.130047 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.130057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.130078 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:11:18.130304 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.130309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.130478 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.130599 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:18.130971 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.132109 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.133773 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.134550 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.134669 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.134826 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.135187 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.135299 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":125,"threads_started":1}
I20260923 18:11:18.135588 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.128691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.27.75.193:42551)
0923 18:11:18.129356 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.129372 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.129753 (+ 381us) server_negotiation.cc:407] Connection header received
0923 18:11:18.129842 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.129847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.130068 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.130264 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:18.130813 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.132070 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.133781 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.134538 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.134639 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.134797 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.135078 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.135172 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":81,"threads_started":1}
I20260923 18:11:18.135883 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.128654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:46357 (local address 127.0.0.1:38758)
0923 18:11:18.129476 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.129486 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.130156 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.130731 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.130733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.130750 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:18.130853 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.130859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.132228 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.132231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.133632 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.133652 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.134773 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.134776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.134864 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.134888 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.134959 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.135031 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":91,"thread_start_us":112,"threads_started":1}
I20260923 18:11:18.138109 3169 tablet_service.cc:1467] Processing CreateTablet for tablet bc40d59caf11437291e5bb1eca8ca60f (DEFAULT_TABLE table=test-table [id=d5c1a5389814450c95e676455f3db16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.138109 3309 tablet_service.cc:1467] Processing CreateTablet for tablet bc40d59caf11437291e5bb1eca8ca60f (DEFAULT_TABLE table=test-table [id=d5c1a5389814450c95e676455f3db16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.138275 3449 tablet_service.cc:1467] Processing CreateTablet for tablet bc40d59caf11437291e5bb1eca8ca60f (DEFAULT_TABLE table=test-table [id=d5c1a5389814450c95e676455f3db16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.139164 3449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc40d59caf11437291e5bb1eca8ca60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.139182 3169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc40d59caf11437291e5bb1eca8ca60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.139182 3309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc40d59caf11437291e5bb1eca8ca60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.147115 3545 tablet_bootstrap.cc:492] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: Bootstrap starting.
I20260923 18:11:18.149171 3545 tablet_bootstrap.cc:654] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.149542 3543 tablet_bootstrap.cc:492] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: Bootstrap starting.
I20260923 18:11:18.150123 3545 log.cc:824] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:18.150880 3544 tablet_bootstrap.cc:492] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: Bootstrap starting.
I20260923 18:11:18.151837 3543 tablet_bootstrap.cc:654] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.152252 3545 tablet_bootstrap.cc:492] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: No bootstrap required, opened a new log
I20260923 18:11:18.152449 3545 ts_tablet_manager.cc:1397] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:11:18.152668 3543 log.cc:824] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:18.152980 3544 tablet_bootstrap.cc:654] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.154068 3544 log.cc:824] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:18.155586 3545 raft_consensus.cc:348] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } }
I20260923 18:11:18.155766 3544 tablet_bootstrap.cc:492] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: No bootstrap required, opened a new log
I20260923 18:11:18.155829 3545 raft_consensus.cc:374] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.155894 3545 raft_consensus.cc:729] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8d0cb1a38f40d89624f5642fad9ebe, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.155942 3544 ts_tablet_manager.cc:1397] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 18:11:18.156394 3545 consensus_queue.cc:260] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } }
I20260923 18:11:18.157021 3375 heartbeater.cc:503] Master 127.27.75.254:40937 was elected leader, sending a full tablet report...
I20260923 18:11:18.157373 3545 ts_tablet_manager.cc:1428] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 18:11:18.157635 3543 tablet_bootstrap.cc:492] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: No bootstrap required, opened a new log
I20260923 18:11:18.157817 3543 ts_tablet_manager.cc:1397] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 18:11:18.159727 3544 raft_consensus.cc:348] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } }
I20260923 18:11:18.160009 3544 raft_consensus.cc:374] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.160064 3544 raft_consensus.cc:729] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23e58d2dfff400581b159a7d171afdf, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.160531 3544 consensus_queue.cc:260] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } }
I20260923 18:11:18.161542 3235 heartbeater.cc:503] Master 127.27.75.254:40937 was elected leader, sending a full tablet report...
I20260923 18:11:18.161603 3544 ts_tablet_manager.cc:1428] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:11:18.161616 3543 raft_consensus.cc:348] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } }
I20260923 18:11:18.161870 3543 raft_consensus.cc:374] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.161937 3543 raft_consensus.cc:729] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aa72c49fbe4414912cf85ae3df635b, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.162356 3543 consensus_queue.cc:260] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } }
I20260923 18:11:18.163034 3515 heartbeater.cc:503] Master 127.27.75.254:40937 was elected leader, sending a full tablet report...
I20260923 18:11:18.163347 3543 ts_tablet_manager.cc:1428] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260923 18:11:18.319119 3516 tablet.cc:2369] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:11:18.359802 3376 tablet.cc:2369] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:11:18.377231 3551 raft_consensus.cc:482] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:11:18.377455 3551 raft_consensus.cc:504] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } }
I20260923 18:11:18.378553 3551 leader_election.cc:288] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551), 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785)
I20260923 18:11:18.382918 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.378922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:36393 (local address 127.27.75.193:42551)
0923 18:11:18.379086 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.379091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.379110 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:11:18.379166 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.379169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.379220 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.379313 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:18.379687 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.380430 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.381697 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.382261 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.382356 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.382399 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.382712 (+ 313us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.382779 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 18:11:18.382931 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.378819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:42551 (local address 127.27.75.195:36393)
0923 18:11:18.378943 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.378958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.379054 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.379351 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.379354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.379383 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:18.379513 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.379520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.380531 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.380535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.381575 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.381584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.382381 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.382385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.382580 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.382613 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.382700 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.382782 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 18:11:18.383260 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.379039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:60175 (local address 127.27.75.194:40785)
0923 18:11:18.379162 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.379166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.379350 (+ 184us) server_negotiation.cc:407] Connection header received
0923 18:11:18.379408 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.379411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.379454 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.379539 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:18.380265 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.380911 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.382164 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.382714 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.382819 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.382866 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.383069 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.383154 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:11:18.383595 3552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.378838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:40785 (local address 127.27.75.195:60175)
0923 18:11:18.379241 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.379253 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.379339 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.379580 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.379582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.379595 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:18.380119 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.380129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.381058 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.381061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.382028 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.382037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.382826 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.382830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.382922 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.382945 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.383393 (+ 448us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.383470 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":119,"threads_started":1}
I20260923 18:11:18.383688 3189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc40d59caf11437291e5bb1eca8ca60f" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23e58d2dfff400581b159a7d171afdf" is_pre_election: true
I20260923 18:11:18.384022 3189 raft_consensus.cc:2454] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 0.
I20260923 18:11:18.384194 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc40d59caf11437291e5bb1eca8ca60f" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" is_pre_election: true
I20260923 18:11:18.384495 3403 leader_election.cc:302] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa72c49fbe4414912cf85ae3df635b, c23e58d2dfff400581b159a7d171afdf; no voters:
I20260923 18:11:18.384518 3329 raft_consensus.cc:2454] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 0.
I20260923 18:11:18.384755 3551 raft_consensus.cc:2781] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:11:18.384843 3551 raft_consensus.cc:482] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:11:18.384888 3551 raft_consensus.cc:3037] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.385980 3551 raft_consensus.cc:504] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } }
I20260923 18:11:18.386298 3551 leader_election.cc:288] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 election: Requested vote from peers c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551), 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785)
I20260923 18:11:18.386637 3189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc40d59caf11437291e5bb1eca8ca60f" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23e58d2dfff400581b159a7d171afdf"
I20260923 18:11:18.386760 3189 raft_consensus.cc:3037] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.386767 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc40d59caf11437291e5bb1eca8ca60f" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe"
I20260923 18:11:18.386878 3329 raft_consensus.cc:3037] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.387737 3189 raft_consensus.cc:2454] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 1.
W20260923 18:11:18.387836 3236 tablet.cc:2369] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:11:18.387835 3329 raft_consensus.cc:2454] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 1.
I20260923 18:11:18.388037 3403 leader_election.cc:302] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa72c49fbe4414912cf85ae3df635b, c23e58d2dfff400581b159a7d171afdf; no voters:
I20260923 18:11:18.388240 3551 raft_consensus.cc:2781] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:18.388422 3551 raft_consensus.cc:686] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 1 LEADER]: Becoming Leader. State: Replica: 68aa72c49fbe4414912cf85ae3df635b, State: Running, Role: LEADER
I20260923 18:11:18.388733 3551 consensus_queue.cc:237] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } }
I20260923 18:11:18.391923 3031 catalog_manager.cc:5495] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b reported cstate change: term changed from 0 to 1, leader changed from <none> to 68aa72c49fbe4414912cf85ae3df635b (127.27.75.195). New cstate: current_term: 1 leader_uuid: "68aa72c49fbe4414912cf85ae3df635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } health_report { overall_health: UNKNOWN } } }
I20260923 18:11:18.429255 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.425606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.27.75.195:46357)
0923 18:11:18.425727 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.425731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.425745 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:18.425923 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.425925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.426016 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.426108 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:11:18.426349 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.426866 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.427704 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.427930 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.428020 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.428293 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:11:18.428808 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:11:18.428932 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.429017 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.429081 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:11:18.436725 3189 raft_consensus.cc:1261] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Refusing update from remote peer 68aa72c49fbe4414912cf85ae3df635b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:18.436724 3329 raft_consensus.cc:1261] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Refusing update from remote peer 68aa72c49fbe4414912cf85ae3df635b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:18.437531 3553 consensus_queue.cc:1035] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:18.437772 3551 consensus_queue.cc:1035] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:18.446105 3558 mvcc.cc:205] Tried to move back new op lower bound from 7332606273265741824 to 7332606273082941440. Current Snapshot: MvccSnapshot[applied={T|T < 7332606273265741824}]
I20260923 18:11:18.453963 3031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:18.454142 3031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:18.456825 3031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa: Sending DeleteTablet for 3 replicas of tablet bc40d59caf11437291e5bb1eca8ca60f
I20260923 18:11:18.457743 3169 tablet_service.cc:1514] Processing DeleteTablet for tablet bc40d59caf11437291e5bb1eca8ca60f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:18 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:34282
I20260923 18:11:18.457796 3309 tablet_service.cc:1514] Processing DeleteTablet for tablet bc40d59caf11437291e5bb1eca8ca60f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:18 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:59320
I20260923 18:11:18.457798 3449 tablet_service.cc:1514] Processing DeleteTablet for tablet bc40d59caf11437291e5bb1eca8ca60f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:18 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:38758
I20260923 18:11:18.458374 3568 tablet_replica.cc:330] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: stopping tablet replica
I20260923 18:11:18.458734 3568 raft_consensus.cc:2229] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:18.459079 3568 raft_consensus.cc:2258] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:18.459067 3031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:11:18.459488 3031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:11:18.459508 3567 tablet_replica.cc:330] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: stopping tablet replica
I20260923 18:11:18.459789 3567 raft_consensus.cc:2229] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:11:18.460142 3567 raft_consensus.cc:2258] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:18.460507 3566 tablet_replica.cc:330] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: stopping tablet replica
I20260923 18:11:18.460795 3566 raft_consensus.cc:2229] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:18.461094 3566 raft_consensus.cc:2258] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:18.461253 3568 ts_tablet_manager.cc:1905] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:18.461747 3567 ts_tablet_manager.cc:1905] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:18.464514 3567 ts_tablet_manager.cc:1918] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:18.464605 3567 log.cc:1196] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/bc40d59caf11437291e5bb1eca8ca60f
I20260923 18:11:18.464687 3449 tablet_service.cc:1467] Processing CreateTablet for tablet 26317b8486774bf69aa29dd513e5b13a (DEFAULT_TABLE table=test-table [id=b843a975fac64c6f9066b371b55f84d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.464898 3567 ts_tablet_manager.cc:1939] T bc40d59caf11437291e5bb1eca8ca60f P 68aa72c49fbe4414912cf85ae3df635b: Deleting consensus metadata
I20260923 18:11:18.464943 3449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26317b8486774bf69aa29dd513e5b13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.465067 3169 tablet_service.cc:1467] Processing CreateTablet for tablet 26317b8486774bf69aa29dd513e5b13a (DEFAULT_TABLE table=test-table [id=b843a975fac64c6f9066b371b55f84d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.465339 3169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26317b8486774bf69aa29dd513e5b13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.465740 3309 tablet_service.cc:1467] Processing CreateTablet for tablet 26317b8486774bf69aa29dd513e5b13a (DEFAULT_TABLE table=test-table [id=b843a975fac64c6f9066b371b55f84d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.465976 3309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26317b8486774bf69aa29dd513e5b13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.466001 3566 ts_tablet_manager.cc:1905] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:18.467492 3544 tablet_bootstrap.cc:492] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf: Bootstrap starting.
I20260923 18:11:18.468303 3543 tablet_bootstrap.cc:492] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b: Bootstrap starting.
I20260923 18:11:18.468443 3544 tablet_bootstrap.cc:654] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.468472 3566 ts_tablet_manager.cc:1918] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:18.468561 3566 log.cc:1196] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/bc40d59caf11437291e5bb1eca8ca60f
I20260923 18:11:18.468835 3566 ts_tablet_manager.cc:1939] T bc40d59caf11437291e5bb1eca8ca60f P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting consensus metadata
I20260923 18:11:18.469209 3543 tablet_bootstrap.cc:654] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.470306 3543 tablet_bootstrap.cc:492] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b: No bootstrap required, opened a new log
I20260923 18:11:18.470386 3543 ts_tablet_manager.cc:1397] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260923 18:11:18.470814 3543 raft_consensus.cc:348] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.470928 3543 raft_consensus.cc:374] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.470960 3543 raft_consensus.cc:729] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aa72c49fbe4414912cf85ae3df635b, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.471007 3017 catalog_manager.cc:4841] TS 68aa72c49fbe4414912cf85ae3df635b (127.27.75.195:46357): tablet bc40d59caf11437291e5bb1eca8ca60f (table test-table [id=d5c1a5389814450c95e676455f3db16f]) successfully deleted
I20260923 18:11:18.471073 3543 consensus_queue.cc:260] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.471309 3543 ts_tablet_manager.cc:1428] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 18:11:18.471469 3016 catalog_manager.cc:4841] TS 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785): tablet bc40d59caf11437291e5bb1eca8ca60f (table test-table [id=d5c1a5389814450c95e676455f3db16f]) successfully deleted
I20260923 18:11:18.472997 3545 tablet_bootstrap.cc:492] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe: Bootstrap starting.
I20260923 18:11:18.464102 3568 ts_tablet_manager.cc:1918] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:18.473392 3568 log.cc:1196] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/wal/wals/bc40d59caf11437291e5bb1eca8ca60f
I20260923 18:11:18.473905 3545 tablet_bootstrap.cc:654] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.474180 3568 ts_tablet_manager.cc:1939] T bc40d59caf11437291e5bb1eca8ca60f P c23e58d2dfff400581b159a7d171afdf: Deleting consensus metadata
I20260923 18:11:18.474939 3545 tablet_bootstrap.cc:492] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe: No bootstrap required, opened a new log
I20260923 18:11:18.475020 3545 ts_tablet_manager.cc:1397] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 18:11:18.475689 3545 raft_consensus.cc:348] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.475801 3545 raft_consensus.cc:374] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.475837 3545 raft_consensus.cc:729] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8d0cb1a38f40d89624f5642fad9ebe, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.475950 3545 consensus_queue.cc:260] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.476011 3544 tablet_bootstrap.cc:492] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf: No bootstrap required, opened a new log
I20260923 18:11:18.476084 3544 ts_tablet_manager.cc:1397] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260923 18:11:18.476203 3545 ts_tablet_manager.cc:1428] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:18.476518 3544 raft_consensus.cc:348] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.476634 3544 raft_consensus.cc:374] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.476672 3544 raft_consensus.cc:729] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23e58d2dfff400581b159a7d171afdf, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.476773 3544 consensus_queue.cc:260] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.477025 3544 ts_tablet_manager.cc:1428] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:18.477252 3018 catalog_manager.cc:4841] TS c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551): tablet bc40d59caf11437291e5bb1eca8ca60f (table test-table [id=d5c1a5389814450c95e676455f3db16f]) successfully deleted
I20260923 18:11:18.561924 3563 raft_consensus.cc:482] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:11:18.562093 3563 raft_consensus.cc:504] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.562542 3563 leader_election.cc:288] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551), 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785)
I20260923 18:11:18.563042 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26317b8486774bf69aa29dd513e5b13a" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" is_pre_election: true
I20260923 18:11:18.563078 3189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26317b8486774bf69aa29dd513e5b13a" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23e58d2dfff400581b159a7d171afdf" is_pre_election: true
I20260923 18:11:18.563189 3329 raft_consensus.cc:2454] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 0.
I20260923 18:11:18.563258 3189 raft_consensus.cc:2454] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 0.
I20260923 18:11:18.563563 3401 leader_election.cc:302] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8d0cb1a38f40d89624f5642fad9ebe, 68aa72c49fbe4414912cf85ae3df635b; no voters:
I20260923 18:11:18.563791 3563 raft_consensus.cc:2781] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:11:18.563874 3563 raft_consensus.cc:482] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:11:18.563918 3563 raft_consensus.cc:3037] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.565025 3563 raft_consensus.cc:504] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.565430 3563 leader_election.cc:288] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 election: Requested vote from peers c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551), 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785)
I20260923 18:11:18.565810 3189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26317b8486774bf69aa29dd513e5b13a" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23e58d2dfff400581b159a7d171afdf"
I20260923 18:11:18.565889 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26317b8486774bf69aa29dd513e5b13a" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe"
I20260923 18:11:18.565950 3189 raft_consensus.cc:3037] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.566008 3329 raft_consensus.cc:3037] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.567071 3189 raft_consensus.cc:2454] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 1.
I20260923 18:11:18.567071 3329 raft_consensus.cc:2454] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 1.
I20260923 18:11:18.567401 3403 leader_election.cc:302] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa72c49fbe4414912cf85ae3df635b, c23e58d2dfff400581b159a7d171afdf; no voters:
I20260923 18:11:18.567569 3563 raft_consensus.cc:2781] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:18.567664 3563 raft_consensus.cc:686] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 1 LEADER]: Becoming Leader. State: Replica: 68aa72c49fbe4414912cf85ae3df635b, State: Running, Role: LEADER
I20260923 18:11:18.567806 3563 consensus_queue.cc:237] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.569525 3029 catalog_manager.cc:5495] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b reported cstate change: term changed from 0 to 1, leader changed from <none> to 68aa72c49fbe4414912cf85ae3df635b (127.27.75.195). New cstate: current_term: 1 leader_uuid: "68aa72c49fbe4414912cf85ae3df635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } health_report { overall_health: HEALTHY } } }
Sep 23 18:11:18 dist-test-slave-wsx5 krb5kdc[2986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187077, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.254@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
I20260923 18:11:18.603854 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.594978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:40937 (local address 127.0.0.1:60504)
0923 18:11:18.595274 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.595286 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.595403 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.595653 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.595655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.596077 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:18.596175 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.596190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.596905 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.596908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.597519 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.597528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.597932 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.597935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.598097 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.598785 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:18.598797 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:18.600404 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:18.602089 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:18.602100 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:18.602107 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:18.602352 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:18.602720 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:18.602723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:18.602725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:18.602825 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:18.603242 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:18.603250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:18.603373 (+ 123us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.603471 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.603648 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":31,"thread_start_us":113,"threads_started":1}
I20260923 18:11:18.603853 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.595054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.27.75.254:40937)
0923 18:11:18.595185 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.595189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.595357 (+ 168us) server_negotiation.cc:407] Connection header received
0923 18:11:18.595486 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.595489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.595529 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.595603 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:18.596299 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.596808 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.597634 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.597834 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.597918 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.600570 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:18.600574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:18.600577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:18.600601 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:18.601983 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:18.602442 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:18.602445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:18.602447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:18.602616 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:18.602911 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:18.602915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:18.602916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:18.603140 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:18.603250 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.603538 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.603699 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":48}
I20260923 18:11:18.608809 3029 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:60504:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 18:11:18.609640 3029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:11:18.614480 3169 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc8cbdb810b4e2fbd006f6917eff475 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ffa32080f6e4e05aff340eb122982d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:11:18.614564 3309 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc8cbdb810b4e2fbd006f6917eff475 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ffa32080f6e4e05aff340eb122982d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:11:18.614611 3449 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc8cbdb810b4e2fbd006f6917eff475 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ffa32080f6e4e05aff340eb122982d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 18:11:18.614924 3169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc8cbdb810b4e2fbd006f6917eff475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.614924 3449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc8cbdb810b4e2fbd006f6917eff475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.614938 3309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc8cbdb810b4e2fbd006f6917eff475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.617575 3545 tablet_bootstrap.cc:492] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe: Bootstrap starting.
I20260923 18:11:18.617728 3544 tablet_bootstrap.cc:492] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf: Bootstrap starting.
I20260923 18:11:18.617888 3543 tablet_bootstrap.cc:492] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: Bootstrap starting.
I20260923 18:11:18.618541 3544 tablet_bootstrap.cc:654] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.618590 3545 tablet_bootstrap.cc:654] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.618681 3543 tablet_bootstrap.cc:654] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.619820 3545 tablet_bootstrap.cc:492] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe: No bootstrap required, opened a new log
I20260923 18:11:18.619894 3545 ts_tablet_manager.cc:1397] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:11:18.620170 3543 tablet_bootstrap.cc:492] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: No bootstrap required, opened a new log
I20260923 18:11:18.620256 3543 ts_tablet_manager.cc:1397] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 18:11:18.620304 3545 raft_consensus.cc:348] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.620419 3545 raft_consensus.cc:374] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.620442 3545 raft_consensus.cc:729] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8d0cb1a38f40d89624f5642fad9ebe, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.620573 3545 consensus_queue.cc:260] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.620612 3543 raft_consensus.cc:348] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.620716 3543 raft_consensus.cc:374] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.620756 3543 raft_consensus.cc:729] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aa72c49fbe4414912cf85ae3df635b, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.620824 3545 ts_tablet_manager.cc:1428] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:11:18.620800 3549 tablet_replica.cc:439] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } }
I20260923 18:11:18.620877 3543 consensus_queue.cc:260] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.620937 3549 tablet_replica.cc:442] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:11:18.621141 3543 ts_tablet_manager.cc:1428] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:11:18.621321 3544 tablet_bootstrap.cc:492] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf: No bootstrap required, opened a new log
I20260923 18:11:18.621392 3544 ts_tablet_manager.cc:1397] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 18:11:18.621148 3563 tablet_replica.cc:439] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } }
I20260923 18:11:18.621596 3563 tablet_replica.cc:442] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:11:18.621799 3544 raft_consensus.cc:348] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.621914 3544 raft_consensus.cc:374] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.621953 3544 raft_consensus.cc:729] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23e58d2dfff400581b159a7d171afdf, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.622069 3544 consensus_queue.cc:260] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.622323 3544 ts_tablet_manager.cc:1428] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:11:18.622298 3550 tablet_replica.cc:439] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } }
I20260923 18:11:18.622407 3550 tablet_replica.cc:442] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:11:18.691372 3563 raft_consensus.cc:482] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:11:18.691515 3563 raft_consensus.cc:504] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.691995 3563 leader_election.cc:288] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551), 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785)
I20260923 18:11:18.692518 3189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc8cbdb810b4e2fbd006f6917eff475" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23e58d2dfff400581b159a7d171afdf" is_pre_election: true
I20260923 18:11:18.692549 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc8cbdb810b4e2fbd006f6917eff475" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" is_pre_election: true
I20260923 18:11:18.692662 3189 raft_consensus.cc:2454] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 0.
I20260923 18:11:18.692698 3329 raft_consensus.cc:2454] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 0.
I20260923 18:11:18.693002 3403 leader_election.cc:302] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa72c49fbe4414912cf85ae3df635b, c23e58d2dfff400581b159a7d171afdf; no voters:
I20260923 18:11:18.693210 3563 raft_consensus.cc:2781] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:11:18.693267 3563 raft_consensus.cc:482] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:11:18.693293 3563 raft_consensus.cc:3037] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.694268 3563 raft_consensus.cc:504] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.694641 3563 leader_election.cc:288] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 election: Requested vote from peers c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551), 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785)
I20260923 18:11:18.695063 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc8cbdb810b4e2fbd006f6917eff475" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe"
I20260923 18:11:18.695050 3189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc8cbdb810b4e2fbd006f6917eff475" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23e58d2dfff400581b159a7d171afdf"
I20260923 18:11:18.695186 3189 raft_consensus.cc:3037] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.695186 3329 raft_consensus.cc:3037] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.696264 3189 raft_consensus.cc:2454] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 1.
I20260923 18:11:18.696264 3329 raft_consensus.cc:2454] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 1.
I20260923 18:11:18.696554 3403 leader_election.cc:302] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa72c49fbe4414912cf85ae3df635b, c23e58d2dfff400581b159a7d171afdf; no voters:
I20260923 18:11:18.696789 3563 raft_consensus.cc:2781] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:18.696897 3563 raft_consensus.cc:686] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [term 1 LEADER]: Becoming Leader. State: Replica: 68aa72c49fbe4414912cf85ae3df635b, State: Running, Role: LEADER
I20260923 18:11:18.697038 3563 consensus_queue.cc:237] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.697718 3553 tablet_replica.cc:439] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: TxnStatusTablet state changed. Reason: New leader 68aa72c49fbe4414912cf85ae3df635b. Latest consensus state: current_term: 1 leader_uuid: "68aa72c49fbe4414912cf85ae3df635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } }
I20260923 18:11:18.697836 3553 tablet_replica.cc:442] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:11:18.698063 3583 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 18:11:18.698246 3583 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 18:11:18.698855 3030 catalog_manager.cc:5495] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b reported cstate change: term changed from 0 to 1, leader changed from <none> to 68aa72c49fbe4414912cf85ae3df635b (127.27.75.195). New cstate: current_term: 1 leader_uuid: "68aa72c49fbe4414912cf85ae3df635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } health_report { overall_health: HEALTHY } } }
I20260923 18:11:18.723721 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.720580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:46357 (local address 127.0.0.1:38784)
0923 18:11:18.720730 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.720746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.720859 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.721107 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.721110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.721145 (+ 35us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:11:18.721249 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.721255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.721944 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.721947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.722576 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.722585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.723010 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.723014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.723094 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.723141 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:11:18.723413 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:11:18.723428 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.723487 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.723585 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260923 18:11:18.723721 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.720724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.27.75.195:46357)
0923 18:11:18.720868 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.720872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.720887 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:11:18.720966 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.720969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.721019 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.721087 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:11:18.721351 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.721844 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.722701 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.722915 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.723001 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.723269 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:11:18.723341 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:11:18.723399 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.723509 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.723574 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 18:11:18.726550 3189 raft_consensus.cc:1261] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Refusing update from remote peer 68aa72c49fbe4414912cf85ae3df635b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:18.726603 3329 raft_consensus.cc:1261] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Refusing update from remote peer 68aa72c49fbe4414912cf85ae3df635b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:18.726991 3553 consensus_queue.cc:1035] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:18.727147 3563 consensus_queue.cc:1035] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:18.728874 3550 tablet_replica.cc:439] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf: TxnStatusTablet state changed. Reason: New leader 68aa72c49fbe4414912cf85ae3df635b. Latest consensus state: current_term: 1 leader_uuid: "68aa72c49fbe4414912cf85ae3df635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } }
I20260923 18:11:18.729002 3550 tablet_replica.cc:442] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:11:18.729027 3549 tablet_replica.cc:439] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe: TxnStatusTablet state changed. Reason: New leader 68aa72c49fbe4414912cf85ae3df635b. Latest consensus state: current_term: 1 leader_uuid: "68aa72c49fbe4414912cf85ae3df635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } }
I20260923 18:11:18.729207 3549 tablet_replica.cc:442] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:11:18.730518 3563 tablet_replica.cc:439] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68aa72c49fbe4414912cf85ae3df635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } }
I20260923 18:11:18.730639 3563 tablet_replica.cc:442] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:11:18.730855 3556 mvcc.cc:205] Tried to move back new op lower bound from 7332606274459328512 to 7332606274344558592. Current Snapshot: MvccSnapshot[applied={T|T < 7332606274459328512}]
I20260923 18:11:18.731712 3550 tablet_replica.cc:439] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68aa72c49fbe4414912cf85ae3df635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } }
I20260923 18:11:18.731817 3550 tablet_replica.cc:442] T 0bc8cbdb810b4e2fbd006f6917eff475 P c23e58d2dfff400581b159a7d171afdf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:11:18.732447 3551 tablet_replica.cc:439] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68aa72c49fbe4414912cf85ae3df635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } }
I20260923 18:11:18.732542 3551 tablet_replica.cc:442] T 0bc8cbdb810b4e2fbd006f6917eff475 P 68aa72c49fbe4414912cf85ae3df635b: This TxnStatusTablet replica's current role is: LEADER
I20260923 18:11:18.734087 3549 tablet_replica.cc:439] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68aa72c49fbe4414912cf85ae3df635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } } }
I20260923 18:11:18.734205 3549 tablet_replica.cc:442] T 0bc8cbdb810b4e2fbd006f6917eff475 P 3e8d0cb1a38f40d89624f5642fad9ebe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 18:11:18.746464 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.743338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:40171 (local address 127.27.75.195:46357)
0923 18:11:18.743458 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.743463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.743690 (+ 227us) server_negotiation.cc:407] Connection header received
0923 18:11:18.743815 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.743818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.743860 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.743919 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:11:18.744131 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.744624 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.745484 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.745700 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.745784 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.746045 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:11:18.746113 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:11:18.746176 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.746255 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.746309 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":48}
I20260923 18:11:18.746464 3589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.743273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:46357 (local address 127.27.75.195:40171)
0923 18:11:18.743608 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.743620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.743731 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.743936 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.743938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.743945 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:11:18.744047 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.744052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.744724 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.744727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.745384 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.745394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.745794 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.745797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.745896 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.745935 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:11:18.746184 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:11:18.746190 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.746247 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.746342 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":34,"thread_start_us":94,"threads_started":1}
I20260923 18:11:18.753085 3329 raft_consensus.cc:1261] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Refusing update from remote peer 68aa72c49fbe4414912cf85ae3df635b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:18.753108 3189 raft_consensus.cc:1261] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Refusing update from remote peer 68aa72c49fbe4414912cf85ae3df635b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:18.753512 3553 consensus_queue.cc:1035] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:18.753648 3551 consensus_queue.cc:1035] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:18.763598 3557 mvcc.cc:205] Tried to move back new op lower bound from 7332606274569371648 to 7332606273815085056. Current Snapshot: MvccSnapshot[applied={T|T < 7332606274569371648}]
I20260923 18:11:18.790764 3031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:18.790916 3031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:18.792989 3031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa: Sending DeleteTablet for 3 replicas of tablet 26317b8486774bf69aa29dd513e5b13a
I20260923 18:11:18.793635 3169 tablet_service.cc:1514] Processing DeleteTablet for tablet 26317b8486774bf69aa29dd513e5b13a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:18 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:34282
I20260923 18:11:18.793678 3309 tablet_service.cc:1514] Processing DeleteTablet for tablet 26317b8486774bf69aa29dd513e5b13a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:18 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:59320
I20260923 18:11:18.793835 3566 tablet_replica.cc:330] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe: stopping tablet replica
I20260923 18:11:18.793835 3568 tablet_replica.cc:330] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf: stopping tablet replica
I20260923 18:11:18.793943 3568 raft_consensus.cc:2229] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:18.793953 3566 raft_consensus.cc:2229] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:18.794070 3566 raft_consensus.cc:2258] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:18.794098 3568 raft_consensus.cc:2258] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:18.794816 3566 ts_tablet_manager.cc:1905] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:18.795037 3568 ts_tablet_manager.cc:1905] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:18.797765 3566 ts_tablet_manager.cc:1918] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:11:18.797797 3568 ts_tablet_manager.cc:1918] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:11:18.797839 3566 log.cc:1196] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/26317b8486774bf69aa29dd513e5b13a
I20260923 18:11:18.797852 3568 log.cc:1196] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/wal/wals/26317b8486774bf69aa29dd513e5b13a
I20260923 18:11:18.798026 3568 ts_tablet_manager.cc:1939] T 26317b8486774bf69aa29dd513e5b13a P c23e58d2dfff400581b159a7d171afdf: Deleting consensus metadata
I20260923 18:11:18.798094 3566 ts_tablet_manager.cc:1939] T 26317b8486774bf69aa29dd513e5b13a P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting consensus metadata
I20260923 18:11:18.798367 3449 tablet_service.cc:1514] Processing DeleteTablet for tablet 26317b8486774bf69aa29dd513e5b13a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:18 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:38758
I20260923 18:11:18.798584 3567 tablet_replica.cc:330] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b: stopping tablet replica
I20260923 18:11:18.798714 3567 raft_consensus.cc:2229] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:11:18.798916 3567 raft_consensus.cc:2258] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:18.798924 3018 catalog_manager.cc:4841] TS c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551): tablet 26317b8486774bf69aa29dd513e5b13a (table test-table [id=b843a975fac64c6f9066b371b55f84d9]) successfully deleted
I20260923 18:11:18.799609 3567 ts_tablet_manager.cc:1905] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:18.802099 3567 ts_tablet_manager.cc:1918] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:11:18.802170 3567 log.cc:1196] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/26317b8486774bf69aa29dd513e5b13a
I20260923 18:11:18.802369 3567 ts_tablet_manager.cc:1939] T 26317b8486774bf69aa29dd513e5b13a P 68aa72c49fbe4414912cf85ae3df635b: Deleting consensus metadata
I20260923 18:11:18.802904 3016 catalog_manager.cc:4841] TS 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785): tablet 26317b8486774bf69aa29dd513e5b13a (table test-table [id=b843a975fac64c6f9066b371b55f84d9]) successfully deleted
I20260923 18:11:18.803224 3017 catalog_manager.cc:4841] TS 68aa72c49fbe4414912cf85ae3df635b (127.27.75.195:46357): tablet 26317b8486774bf69aa29dd513e5b13a (table test-table [id=b843a975fac64c6f9066b371b55f84d9]) successfully deleted
Sep 23 18:11:18 dist-test-slave-wsx5 krb5kdc[2986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 18:11:18 dist-test-slave-wsx5 krb5kdc[2986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:18 dist-test-slave-wsx5 krb5kdc[2986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:11:18.824353 3081 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:18.817093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.27.75.254:40937)
0923 18:11:18.817281 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.817286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.817418 (+ 132us) server_negotiation.cc:407] Connection header received
0923 18:11:18.817537 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.817541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.817599 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.817703 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:18.818403 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.819261 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.820262 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.820619 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.820738 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.824210 (+ 3472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60520: BlockingRecv error: recv got EOF from 127.0.0.1:60520 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 23 18:11:18 dist-test-slave-wsx5 krb5kdc[2986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM
I20260923 18:11:18.835228 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.825815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.27.75.254:40937)
0923 18:11:18.825934 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.825938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.826050 (+ 112us) server_negotiation.cc:407] Connection header received
0923 18:11:18.826200 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.826203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.826250 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.826319 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:18.827063 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.827576 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.828701 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.828899 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.828969 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.831882 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:18.831885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:18.831887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:18.831917 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:18.833537 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:18.834012 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:18.834016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:18.834017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:18.834112 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:18.834365 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:18.834368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:18.834370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:18.834569 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:18.834694 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.834930 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.835045 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 18:11:18.838085 3031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:11:18.838589 3031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:11:18.843616 3169 tablet_service.cc:1467] Processing CreateTablet for tablet f7b7a40a16164e1c9c57b5273497c765 (DEFAULT_TABLE table=test-table [id=6d3740154f764baf95e992c2abcbf1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.843612 3309 tablet_service.cc:1467] Processing CreateTablet for tablet f7b7a40a16164e1c9c57b5273497c765 (DEFAULT_TABLE table=test-table [id=6d3740154f764baf95e992c2abcbf1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.843920 3309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b7a40a16164e1c9c57b5273497c765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.843922 3169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b7a40a16164e1c9c57b5273497c765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.846081 3544 tablet_bootstrap.cc:492] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf: Bootstrap starting.
I20260923 18:11:18.846314 3449 tablet_service.cc:1467] Processing CreateTablet for tablet f7b7a40a16164e1c9c57b5273497c765 (DEFAULT_TABLE table=test-table [id=6d3740154f764baf95e992c2abcbf1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.846489 3545 tablet_bootstrap.cc:492] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe: Bootstrap starting.
I20260923 18:11:18.846583 3449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b7a40a16164e1c9c57b5273497c765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.847014 3544 tablet_bootstrap.cc:654] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.847412 3545 tablet_bootstrap.cc:654] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.848582 3545 tablet_bootstrap.cc:492] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe: No bootstrap required, opened a new log
I20260923 18:11:18.848668 3545 ts_tablet_manager.cc:1397] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 18:11:18.848838 3543 tablet_bootstrap.cc:492] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b: Bootstrap starting.
I20260923 18:11:18.849107 3545 raft_consensus.cc:348] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.849260 3545 raft_consensus.cc:374] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.849291 3545 raft_consensus.cc:729] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8d0cb1a38f40d89624f5642fad9ebe, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.849414 3545 consensus_queue.cc:260] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.849663 3545 ts_tablet_manager.cc:1428] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 18:11:18.849668 3543 tablet_bootstrap.cc:654] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.850860 3544 tablet_bootstrap.cc:492] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf: No bootstrap required, opened a new log
I20260923 18:11:18.850926 3544 ts_tablet_manager.cc:1397] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260923 18:11:18.851279 3544 raft_consensus.cc:348] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.851364 3544 raft_consensus.cc:374] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.851392 3544 raft_consensus.cc:729] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23e58d2dfff400581b159a7d171afdf, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.851516 3544 consensus_queue.cc:260] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.851780 3544 ts_tablet_manager.cc:1428] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:18.852778 3543 tablet_bootstrap.cc:492] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b: No bootstrap required, opened a new log
I20260923 18:11:18.852880 3543 ts_tablet_manager.cc:1397] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 18:11:18.853308 3543 raft_consensus.cc:348] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.853420 3543 raft_consensus.cc:374] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.853449 3543 raft_consensus.cc:729] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aa72c49fbe4414912cf85ae3df635b, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.853580 3543 consensus_queue.cc:260] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.853845 3543 ts_tablet_manager.cc:1428] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 18:11:18.877885 3563 raft_consensus.cc:482] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:11:18.878013 3563 raft_consensus.cc:504] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.878433 3563 leader_election.cc:288] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551), 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785)
I20260923 18:11:18.878839 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b7a40a16164e1c9c57b5273497c765" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" is_pre_election: true
I20260923 18:11:18.878862 3189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b7a40a16164e1c9c57b5273497c765" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23e58d2dfff400581b159a7d171afdf" is_pre_election: true
I20260923 18:11:18.878962 3329 raft_consensus.cc:2454] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 0.
I20260923 18:11:18.879001 3189 raft_consensus.cc:2454] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 0.
I20260923 18:11:18.879283 3401 leader_election.cc:302] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8d0cb1a38f40d89624f5642fad9ebe, 68aa72c49fbe4414912cf85ae3df635b; no voters:
I20260923 18:11:18.879444 3563 raft_consensus.cc:2781] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:11:18.879508 3563 raft_consensus.cc:482] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:11:18.879544 3563 raft_consensus.cc:3037] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.880467 3563 raft_consensus.cc:504] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.880776 3563 leader_election.cc:288] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 election: Requested vote from peers c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551), 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785)
I20260923 18:11:18.881182 3189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b7a40a16164e1c9c57b5273497c765" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c23e58d2dfff400581b159a7d171afdf"
I20260923 18:11:18.881178 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b7a40a16164e1c9c57b5273497c765" candidate_uuid: "68aa72c49fbe4414912cf85ae3df635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe"
I20260923 18:11:18.881307 3189 raft_consensus.cc:3037] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.881307 3329 raft_consensus.cc:3037] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.882401 3189 raft_consensus.cc:2454] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 1.
I20260923 18:11:18.882401 3329 raft_consensus.cc:2454] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa72c49fbe4414912cf85ae3df635b in term 1.
I20260923 18:11:18.882697 3403 leader_election.cc:302] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa72c49fbe4414912cf85ae3df635b, c23e58d2dfff400581b159a7d171afdf; no voters:
I20260923 18:11:18.882927 3563 raft_consensus.cc:2781] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:18.883026 3563 raft_consensus.cc:686] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 1 LEADER]: Becoming Leader. State: Replica: 68aa72c49fbe4414912cf85ae3df635b, State: Running, Role: LEADER
I20260923 18:11:18.883173 3563 consensus_queue.cc:237] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.885003 3029 catalog_manager.cc:5495] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b reported cstate change: term changed from 0 to 1, leader changed from <none> to 68aa72c49fbe4414912cf85ae3df635b (127.27.75.195). New cstate: current_term: 1 leader_uuid: "68aa72c49fbe4414912cf85ae3df635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } health_report { overall_health: HEALTHY } } }
I20260923 18:11:18.897328 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.894375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.27.75.195:46357)
0923 18:11:18.894496 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.894499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.894513 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:18.894599 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.894602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.894650 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.894711 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:11:18.894935 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.895436 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.896230 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.896449 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.896585 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.896843 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:11:18.896915 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:11:18.896971 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.897155 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.897212 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 18:11:18.899159 3189 raft_consensus.cc:1261] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Refusing update from remote peer 68aa72c49fbe4414912cf85ae3df635b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:18.899227 3329 raft_consensus.cc:1261] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Refusing update from remote peer 68aa72c49fbe4414912cf85ae3df635b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:18.899576 3563 consensus_queue.cc:1035] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:18.899736 3551 consensus_queue.cc:1035] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:18.906854 3030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:18.906992 3030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:18.909005 3030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa: Sending DeleteTablet for 3 replicas of tablet f7b7a40a16164e1c9c57b5273497c765
I20260923 18:11:18.909564 3169 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b7a40a16164e1c9c57b5273497c765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:18 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:34282
I20260923 18:11:18.909601 3449 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b7a40a16164e1c9c57b5273497c765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:18 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:38758
I20260923 18:11:18.909652 3309 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b7a40a16164e1c9c57b5273497c765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:18 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:59320
I20260923 18:11:18.909754 3568 tablet_replica.cc:330] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf: stopping tablet replica
I20260923 18:11:18.909830 3566 tablet_replica.cc:330] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe: stopping tablet replica
I20260923 18:11:18.909850 3567 tablet_replica.cc:330] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b: stopping tablet replica
I20260923 18:11:18.909865 3568 raft_consensus.cc:2229] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:18.909927 3566 raft_consensus.cc:2229] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:18.909991 3568 raft_consensus.cc:2258] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:18.909981 3567 raft_consensus.cc:2229] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:11:18.910032 3566 raft_consensus.cc:2258] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:18.910171 3567 raft_consensus.cc:2258] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:18.910356 3030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 18:11:18.910733 3568 ts_tablet_manager.cc:1905] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 18:11:18.910771 3030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:11:18.910856 3567 ts_tablet_manager.cc:1905] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:18.912881 3567 ts_tablet_manager.cc:1918] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:18.912962 3567 log.cc:1196] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/f7b7a40a16164e1c9c57b5273497c765
I20260923 18:11:18.913229 3567 ts_tablet_manager.cc:1939] T f7b7a40a16164e1c9c57b5273497c765 P 68aa72c49fbe4414912cf85ae3df635b: Deleting consensus metadata
I20260923 18:11:18.913933 3017 catalog_manager.cc:4841] TS 68aa72c49fbe4414912cf85ae3df635b (127.27.75.195:46357): tablet f7b7a40a16164e1c9c57b5273497c765 (table test-table [id=6d3740154f764baf95e992c2abcbf1cb]) successfully deleted
I20260923 18:11:18.914286 3566 ts_tablet_manager.cc:1905] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:18.914359 3568 ts_tablet_manager.cc:1918] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:18.914440 3568 log.cc:1196] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-0/wal/wals/f7b7a40a16164e1c9c57b5273497c765
I20260923 18:11:18.914685 3568 ts_tablet_manager.cc:1939] T f7b7a40a16164e1c9c57b5273497c765 P c23e58d2dfff400581b159a7d171afdf: Deleting consensus metadata
I20260923 18:11:18.915324 3018 catalog_manager.cc:4841] TS c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551): tablet f7b7a40a16164e1c9c57b5273497c765 (table test-table [id=6d3740154f764baf95e992c2abcbf1cb]) successfully deleted
I20260923 18:11:18.915841 3169 tablet_service.cc:1467] Processing CreateTablet for tablet dfebe5c678e04086962980bd371e8884 (DEFAULT_TABLE table=test-table [id=7ca2c8982c7542b9875858b11437ff89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.916082 3169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfebe5c678e04086962980bd371e8884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.916558 3566 ts_tablet_manager.cc:1918] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:18.916625 3566 log.cc:1196] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/f7b7a40a16164e1c9c57b5273497c765
I20260923 18:11:18.916803 3566 ts_tablet_manager.cc:1939] T f7b7a40a16164e1c9c57b5273497c765 P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting consensus metadata
I20260923 18:11:18.917063 3309 tablet_service.cc:1467] Processing CreateTablet for tablet dfebe5c678e04086962980bd371e8884 (DEFAULT_TABLE table=test-table [id=7ca2c8982c7542b9875858b11437ff89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.917303 3309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfebe5c678e04086962980bd371e8884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.917466 3016 catalog_manager.cc:4841] TS 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785): tablet f7b7a40a16164e1c9c57b5273497c765 (table test-table [id=6d3740154f764baf95e992c2abcbf1cb]) successfully deleted
I20260923 18:11:18.917541 3449 tablet_service.cc:1467] Processing CreateTablet for tablet dfebe5c678e04086962980bd371e8884 (DEFAULT_TABLE table=test-table [id=7ca2c8982c7542b9875858b11437ff89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:18.917766 3449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfebe5c678e04086962980bd371e8884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:18.918088 3544 tablet_bootstrap.cc:492] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf: Bootstrap starting.
I20260923 18:11:18.919085 3544 tablet_bootstrap.cc:654] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.919293 3545 tablet_bootstrap.cc:492] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe: Bootstrap starting.
I20260923 18:11:18.919919 3543 tablet_bootstrap.cc:492] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b: Bootstrap starting.
I20260923 18:11:18.920094 3545 tablet_bootstrap.cc:654] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.920282 3544 tablet_bootstrap.cc:492] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf: No bootstrap required, opened a new log
I20260923 18:11:18.920364 3544 ts_tablet_manager.cc:1397] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 18:11:18.920652 3543 tablet_bootstrap.cc:654] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:18.920792 3544 raft_consensus.cc:348] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.920905 3544 raft_consensus.cc:374] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.920936 3544 raft_consensus.cc:729] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23e58d2dfff400581b159a7d171afdf, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.921046 3544 consensus_queue.cc:260] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.921337 3544 ts_tablet_manager.cc:1428] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:18.921684 3545 tablet_bootstrap.cc:492] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe: No bootstrap required, opened a new log
I20260923 18:11:18.921770 3545 ts_tablet_manager.cc:1397] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260923 18:11:18.922147 3545 raft_consensus.cc:348] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.922260 3545 raft_consensus.cc:374] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.922300 3545 raft_consensus.cc:729] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8d0cb1a38f40d89624f5642fad9ebe, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.922415 3545 consensus_queue.cc:260] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.922714 3545 ts_tablet_manager.cc:1428] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 18:11:18.923200 3543 tablet_bootstrap.cc:492] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b: No bootstrap required, opened a new log
I20260923 18:11:18.923293 3543 ts_tablet_manager.cc:1397] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 18:11:18.923705 3543 raft_consensus.cc:348] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.923818 3543 raft_consensus.cc:374] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:18.923857 3543 raft_consensus.cc:729] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aa72c49fbe4414912cf85ae3df635b, State: Initialized, Role: FOLLOWER
I20260923 18:11:18.923990 3543 consensus_queue.cc:260] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.924269 3543 ts_tablet_manager.cc:1428] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:11:18.966625 3550 raft_consensus.cc:482] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:11:18.966771 3550 raft_consensus.cc:504] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.967917 3550 leader_election.cc:288] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785), 68aa72c49fbe4414912cf85ae3df635b (127.27.75.195:46357)
I20260923 18:11:18.972229 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.968179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:46357 (local address 127.27.75.193:42445)
0923 18:11:18.968424 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.968439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.968557 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.968831 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.968835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.968860 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:18.969007 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.969014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.969936 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.969940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.970885 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.970895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.971716 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.971721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.971823 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.971850 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.971937 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.972015 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":94,"threads_started":1}
I20260923 18:11:18.972263 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.968262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:42445 (local address 127.27.75.195:46357)
0923 18:11:18.968424 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.968429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.968513 (+ 84us) server_negotiation.cc:407] Connection header received
0923 18:11:18.968661 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.968665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.968724 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.968801 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:18.969187 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.969800 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.971065 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.971575 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.971679 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.971735 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.971995 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.972092 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 18:11:18.972712 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.968213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:40785 (local address 127.27.75.193:57565)
0923 18:11:18.968786 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:18.968799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:18.968881 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:18.969186 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:18.969190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:18.969209 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:18.969327 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.969333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.970272 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.970274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.971197 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:18.971207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.972077 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.972081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:18.972197 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.972221 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:11:18.972458 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 18:11:18.972529 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":59,"threads_started":1}
I20260923 18:11:18.972754 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:18.968505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:57565 (local address 127.27.75.194:40785)
0923 18:11:18.968764 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:18.968767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:18.968868 (+ 101us) server_negotiation.cc:407] Connection header received
0923 18:11:18.968974 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:18.968977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:18.969027 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:18.969109 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:18.969450 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.970131 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.971337 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:18.971948 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:18.972072 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:18.972121 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:18.972496 (+ 375us) server_negotiation.cc:299] Negotiation successful
0923 18:11:18.972622 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260923 18:11:18.972867 3469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfebe5c678e04086962980bd371e8884" candidate_uuid: "c23e58d2dfff400581b159a7d171afdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aa72c49fbe4414912cf85ae3df635b" is_pre_election: true
I20260923 18:11:18.973171 3469 raft_consensus.cc:2454] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c23e58d2dfff400581b159a7d171afdf in term 0.
I20260923 18:11:18.973251 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfebe5c678e04086962980bd371e8884" candidate_uuid: "c23e58d2dfff400581b159a7d171afdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" is_pre_election: true
I20260923 18:11:18.973402 3329 raft_consensus.cc:2454] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c23e58d2dfff400581b159a7d171afdf in term 0.
I20260923 18:11:18.973649 3122 leader_election.cc:302] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa72c49fbe4414912cf85ae3df635b, c23e58d2dfff400581b159a7d171afdf; no voters:
I20260923 18:11:18.973929 3550 raft_consensus.cc:2781] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:11:18.973989 3550 raft_consensus.cc:482] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:11:18.974030 3550 raft_consensus.cc:3037] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.975013 3550 raft_consensus.cc:504] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.975348 3550 leader_election.cc:288] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [CANDIDATE]: Term 1 election: Requested vote from peers 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785), 68aa72c49fbe4414912cf85ae3df635b (127.27.75.195:46357)
I20260923 18:11:18.975776 3469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfebe5c678e04086962980bd371e8884" candidate_uuid: "c23e58d2dfff400581b159a7d171afdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aa72c49fbe4414912cf85ae3df635b"
I20260923 18:11:18.975785 3329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfebe5c678e04086962980bd371e8884" candidate_uuid: "c23e58d2dfff400581b159a7d171afdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe"
I20260923 18:11:18.975905 3329 raft_consensus.cc:3037] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.975905 3469 raft_consensus.cc:3037] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:18.977012 3329 raft_consensus.cc:2454] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c23e58d2dfff400581b159a7d171afdf in term 1.
I20260923 18:11:18.977053 3469 raft_consensus.cc:2454] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c23e58d2dfff400581b159a7d171afdf in term 1.
I20260923 18:11:18.977425 3121 leader_election.cc:302] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8d0cb1a38f40d89624f5642fad9ebe, c23e58d2dfff400581b159a7d171afdf; no voters:
I20260923 18:11:18.977658 3550 raft_consensus.cc:2781] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:18.977897 3550 raft_consensus.cc:686] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 1 LEADER]: Becoming Leader. State: Replica: c23e58d2dfff400581b159a7d171afdf, State: Running, Role: LEADER
I20260923 18:11:18.978163 3550 consensus_queue.cc:237] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } }
I20260923 18:11:18.980472 3031 catalog_manager.cc:5495] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf reported cstate change: term changed from 0 to 1, leader changed from <none> to c23e58d2dfff400581b159a7d171afdf (127.27.75.193). New cstate: current_term: 1 leader_uuid: "c23e58d2dfff400581b159a7d171afdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c23e58d2dfff400581b159a7d171afdf" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 } health_report { overall_health: UNKNOWN } } }
I20260923 18:11:19.006704 3620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:19.003176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.27.75.193:42551)
0923 18:11:19.003432 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:19.003438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:19.003450 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:11:19.003562 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:19.003564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:19.003618 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:19.003719 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:11:19.003885 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:19.004397 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:19.005253 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:19.005550 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:19.005643 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:19.005898 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:11:19.006334 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:11:19.006415 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:19.006525 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 18:11:19.006576 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
I20260923 18:11:19.017163 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:19.013811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:51585 (local address 127.27.75.195:46357)
0923 18:11:19.013953 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:19.013958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:19.014115 (+ 157us) server_negotiation.cc:407] Connection header received
0923 18:11:19.014243 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:19.014246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:19.014299 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:19.014360 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:11:19.014669 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:19.015191 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:19.016077 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:19.016293 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:19.016382 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:19.016689 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:11:19.016765 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:11:19.016822 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:19.016917 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 18:11:19.016968 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:11:19.017180 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:19.013716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:46357 (local address 127.27.75.193:51585)
0923 18:11:19.014029 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:19.014041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:19.014147 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:19.014376 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:19.014379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:19.014386 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:11:19.014561 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:19.014568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:19.015290 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:19.015294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:19.015953 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:19.015962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:19.016405 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:19.016409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:19.016510 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:19.016557 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:11:19.016838 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:11:19.016845 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:11:19.016907 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 18:11:19.016992 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":103,"threads_started":1}
I20260923 18:11:19.022974 3329 raft_consensus.cc:1261] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Refusing update from remote peer c23e58d2dfff400581b159a7d171afdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:19.023113 3469 raft_consensus.cc:1261] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Refusing update from remote peer c23e58d2dfff400581b159a7d171afdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:19.023411 3618 consensus_queue.cc:1035] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8d0cb1a38f40d89624f5642fad9ebe" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 40785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:19.023584 3550 consensus_queue.cc:1035] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "68aa72c49fbe4414912cf85ae3df635b" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 46357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:19.040575 3589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:19.037254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:42551 (local address 127.27.75.195:37883)
0923 18:11:19.037483 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:19.037499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:19.037585 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:19.037854 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:19.037857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:19.037866 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 18:11:19.037981 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:19.037987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:19.038780 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:19.038783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:19.039379 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:19.039388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:19.039914 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:19.039917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:19.040002 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:19.040038 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 18:11:19.040317 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 18:11:19.040324 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 18:11:19.040379 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 18:11:19.040459 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260923 18:11:19.040644 3620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:19.037295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:37883 (local address 127.27.75.193:42551)
0923 18:11:19.037404 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:19.037408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:19.037562 (+ 154us) server_negotiation.cc:407] Connection header received
0923 18:11:19.037700 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:19.037703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:19.037771 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:19.037837 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:11:19.038136 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:19.038682 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:19.039516 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:19.039821 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:19.039911 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:19.040150 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:11:19.040239 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:11:19.040300 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:19.040399 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 18:11:19.040458 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 18:11:19.060256 3031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:19.060416 3031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:19.062511 3031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edd08fb4e557423492e25a12388421fa: Sending DeleteTablet for 3 replicas of tablet dfebe5c678e04086962980bd371e8884
I20260923 18:11:19.063140 3309 tablet_service.cc:1514] Processing DeleteTablet for tablet dfebe5c678e04086962980bd371e8884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:19 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:59320
I20260923 18:11:19.063189 3169 tablet_service.cc:1514] Processing DeleteTablet for tablet dfebe5c678e04086962980bd371e8884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:19 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:34282
I20260923 18:11:19.063346 3566 tablet_replica.cc:330] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe: stopping tablet replica
I20260923 18:11:19.063359 3568 tablet_replica.cc:330] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf: stopping tablet replica
I20260923 18:11:19.063449 3568 raft_consensus.cc:2229] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:11:19.063450 3566 raft_consensus.cc:2229] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:19.063565 3566 raft_consensus.cc:2258] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:19.063764 3568 raft_consensus.cc:2258] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:19.064103 3449 tablet_service.cc:1514] Processing DeleteTablet for tablet dfebe5c678e04086962980bd371e8884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:19 UTC) from {username='oryx', principal='oryx/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:38758
I20260923 18:11:19.064296 3567 tablet_replica.cc:330] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b: stopping tablet replica
I20260923 18:11:19.064397 3567 raft_consensus.cc:2229] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:19.064493 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 3104
I20260923 18:11:19.064538 3568 ts_tablet_manager.cc:1905] T dfebe5c678e04086962980bd371e8884 P c23e58d2dfff400581b159a7d171afdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:19.064532 3567 raft_consensus.cc:2258] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:19.066090 3567 ts_tablet_manager.cc:1905] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:19.066383 3566 ts_tablet_manager.cc:1905] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:19.068603 3567 ts_tablet_manager.cc:1918] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:11:19.068682 3567 log.cc:1196] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/dfebe5c678e04086962980bd371e8884
I20260923 18:11:19.068763 3566 ts_tablet_manager.cc:1918] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 18:11:19.068830 3566 log.cc:1196] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/dfebe5c678e04086962980bd371e8884
I20260923 18:11:19.068897 3567 ts_tablet_manager.cc:1939] T dfebe5c678e04086962980bd371e8884 P 68aa72c49fbe4414912cf85ae3df635b: Deleting consensus metadata
I20260923 18:11:19.069048 3566 ts_tablet_manager.cc:1939] T dfebe5c678e04086962980bd371e8884 P 3e8d0cb1a38f40d89624f5642fad9ebe: Deleting consensus metadata
I20260923 18:11:19.069689 3017 catalog_manager.cc:4841] TS 68aa72c49fbe4414912cf85ae3df635b (127.27.75.195:46357): tablet dfebe5c678e04086962980bd371e8884 (table test-table [id=7ca2c8982c7542b9875858b11437ff89]) successfully deleted
I20260923 18:11:19.070071 3016 catalog_manager.cc:4841] TS 3e8d0cb1a38f40d89624f5642fad9ebe (127.27.75.194:40785): tablet dfebe5c678e04086962980bd371e8884 (table test-table [id=7ca2c8982c7542b9875858b11437ff89]) successfully deleted
W20260923 18:11:19.071480 3018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:42551 (error 108)
I20260923 18:11:19.071695 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 3244
I20260923 18:11:19.076470 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:19.072259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:42551 (local address 127.0.0.1:34294)
0923 18:11:19.076254 (+ 3995us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:19.076355 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3898,"thread_start_us":50,"threads_started":1}
W20260923 18:11:19.076689 3018 catalog_manager.cc:4584] TS c23e58d2dfff400581b159a7d171afdf (127.27.75.193:42551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfebe5c678e04086962980bd371e8884: Network error: Client connection negotiation failed: client connection to 127.27.75.193:42551: connect: Connection refused (error 111)
I20260923 18:11:19.077549 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 3384
I20260923 18:11:19.083382 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 2999
2026-09-23T18:11:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2990 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:11:19 dist-test-slave-wsx5 krb5kdc[3632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:11:19 dist-test-slave-wsx5 krb5kdc[3632](info): set up 2 sockets
Sep 23 18:11:19 dist-test-slave-wsx5 krb5kdc[3632](info): commencing operation
krb5kdc: starting...
W20260923 18:11:20.043783 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:11:20 dist-test-slave-wsx5 krb5kdc[3632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:11:20.170035 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34703
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:34703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:20.281046 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:20.281345 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:20.281404 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:20.285562 3645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:20.285650 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:20.285676 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:20.285697 3645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:20.285717 3645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:20.291218 3645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:34703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:20.292325 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:20.293396 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:20.302459 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:20.302531 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:20.302623 3645 server_base.cc:1034] running on GCE node
W20260923 18:11:20.302523 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:20.303382 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:20.304592 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:20.305771 3645 hybrid_clock.cc:648] HybridClock initialized: now 1790187080305744 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T18:11:20Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1235 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:11:20 dist-test-slave-wsx5 krb5kdc[3658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:11:20 dist-test-slave-wsx5 krb5kdc[3658](info): set up 2 sockets
Sep 23 18:11:20 dist-test-slave-wsx5 krb5kdc[3658](info): commencing operation
krb5kdc: starting...
W20260923 18:11:21.275714 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:11:21 dist-test-slave-wsx5 krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:11:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:11:21.396926 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34473
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:21.507720 3671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:21.508015 3671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:21.508056 3671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:21.512223 3671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:21.512290 3671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:21.512307 3671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:21.512324 3671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:21.512338 3671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:21.517892 3671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:34473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:21.518962 3671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:21.519985 3671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:11:21.529284 3671 server_base.cc:1034] running on GCE node
W20260923 18:11:21.529155 3679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:21.529150 3676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:21.529194 3677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:21.530107 3671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:21.531287 3671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:21.532505 3671 hybrid_clock.cc:648] HybridClock initialized: now 1790187081532486 us; error 33 us; skew 500 ppm
Sep 23 18:11:21 dist-test-slave-wsx5 krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187081, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:21.539915 3671 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:11:21.540939 3671 webserver.cc:460] Webserver started at http://127.27.75.254:37011/ using document root <none> and password file <none>
I20260923 18:11:21.541535 3671 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:21.541607 3671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:21.541782 3671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:21.543902 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "4a49256b7f4c4821aa9f88d621b51442"
format_stamp: "Formatted at 2026-09-23 18:11:21 on dist-test-slave-wsx5"
server_key: "9468ac2270ca9bc72145ef53dba14ee4"
server_key_iv: "fe4cc001ae2d11c2b9a7012245279f51"
server_key_version: "encryptionkey@0"
I20260923 18:11:21.544476 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "4a49256b7f4c4821aa9f88d621b51442"
format_stamp: "Formatted at 2026-09-23 18:11:21 on dist-test-slave-wsx5"
server_key: "9468ac2270ca9bc72145ef53dba14ee4"
server_key_iv: "fe4cc001ae2d11c2b9a7012245279f51"
server_key_version: "encryptionkey@0"
I20260923 18:11:21.548398 3671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:11:21.551083 3686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:21.552445 3671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:11:21.552567 3671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "4a49256b7f4c4821aa9f88d621b51442"
format_stamp: "Formatted at 2026-09-23 18:11:21 on dist-test-slave-wsx5"
server_key: "9468ac2270ca9bc72145ef53dba14ee4"
server_key_iv: "fe4cc001ae2d11c2b9a7012245279f51"
server_key_version: "encryptionkey@0"
I20260923 18:11:21.552672 3671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:21.563815 3671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:21.567106 3671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:21.567297 3671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:21.574092 3671 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:34473
I20260923 18:11:21.574137 3738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:34473 every 8 connection(s)
I20260923 18:11:21.575059 3671 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:21.578095 3739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:21.583518 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 3671
I20260923 18:11:21.583673 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:21.583963 27951 external_mini_cluster.cc:1428] Setting key be4286085ae0b1ed0b6fc579f18b64ce
I20260923 18:11:21.584482 3739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442: Bootstrap starting.
I20260923 18:11:21.587023 3739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:21.587785 3739 log.cc:824] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:21.589725 3739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442: No bootstrap required, opened a new log
Sep 23 18:11:21 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:21.593636 3739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } }
I20260923 18:11:21.593905 3739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:21.593971 3739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a49256b7f4c4821aa9f88d621b51442, State: Initialized, Role: FOLLOWER
I20260923 18:11:21.594488 3739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } }
I20260923 18:11:21.594660 3739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:21.594730 3739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:21.594830 3739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:21.595755 3739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } }
I20260923 18:11:21.596122 3739 leader_election.cc:302] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a49256b7f4c4821aa9f88d621b51442; no voters:
I20260923 18:11:21.596551 3739 leader_election.cc:288] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:21.596712 3744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:21.596935 3744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 1 LEADER]: Becoming Leader. State: Replica: 4a49256b7f4c4821aa9f88d621b51442, State: Running, Role: LEADER
I20260923 18:11:21.597338 3744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } }
I20260923 18:11:21.598006 3739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:21.599056 3746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a49256b7f4c4821aa9f88d621b51442. Latest consensus state: current_term: 1 leader_uuid: "4a49256b7f4c4821aa9f88d621b51442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } } }
I20260923 18:11:21.599190 3746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:21.599452 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:21.585500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.27.75.254:34473)
0923 18:11:21.585910 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:21.585929 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:21.585965 (+ 36us) server_negotiation.cc:407] Connection header received
0923 18:11:21.586622 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:21.586640 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:21.586995 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:21.587339 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:21.588165 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:21.589176 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:21.590062 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:21.590535 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:21.590692 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:21.593474 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:21.593524 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:21.593539 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:21.593571 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:21.595253 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:21.595688 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:21.595694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:21.595700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:21.595818 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:21.596253 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:21.596257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:21.596259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:21.596834 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:21.597831 (+ 997us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:21.598055 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 18:11:21.598369 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":125,"threads_started":1}
I20260923 18:11:21.599515 3745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a49256b7f4c4821aa9f88d621b51442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } } }
I20260923 18:11:21.599586 3745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [sys.catalog]: This master's current role is: LEADER
W20260923 18:11:21.602843 3759 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:11:21.602910 3759 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:11:21.602986 3753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:21.603688 3753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:21.609377 3753 catalog_manager.cc:1347] Generated new cluster ID: 7f777d0ef7874189bf92a1f094152424
I20260923 18:11:21.609458 3753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:21.623610 3753 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:21.624608 3753 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:21.636164 3753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442: Generated new TSK 0
I20260923 18:11:21.636889 3753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:11:21.673110 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34473
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:21.793754 3765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:21.794061 3765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:21.794124 3765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:21.798465 3765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:21.798543 3765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:21.798633 3765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:21.804287 3765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:21.805426 3765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:21.806455 3765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:21.816062 3771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:21.816054 3773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:21.816054 3770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:21.816291 3765 server_base.cc:1034] running on GCE node
I20260923 18:11:21.816820 3765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:21.817564 3765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:21.818727 3765 hybrid_clock.cc:648] HybridClock initialized: now 1790187081818706 us; error 39 us; skew 500 ppm
Sep 23 18:11:21 dist-test-slave-wsx5 krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187081, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:21.821460 3765 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:11:21.822376 3765 webserver.cc:460] Webserver started at http://127.27.75.193:45029/ using document root <none> and password file <none>
I20260923 18:11:21.822919 3765 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:21.822984 3765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:21.823185 3765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:21.825273 3765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "3fd8de4b2b3140f2998a1f72d382b894"
format_stamp: "Formatted at 2026-09-23 18:11:21 on dist-test-slave-wsx5"
server_key: "8013ae1bdc4c27a020db167067281a20"
server_key_iv: "255e09945d1b539f5eee517befd368a4"
server_key_version: "encryptionkey@0"
I20260923 18:11:21.825832 3765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "3fd8de4b2b3140f2998a1f72d382b894"
format_stamp: "Formatted at 2026-09-23 18:11:21 on dist-test-slave-wsx5"
server_key: "8013ae1bdc4c27a020db167067281a20"
server_key_iv: "255e09945d1b539f5eee517befd368a4"
server_key_version: "encryptionkey@0"
I20260923 18:11:21.829650 3765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:11:21.832399 3780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:21.833937 3765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:21.834070 3765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "3fd8de4b2b3140f2998a1f72d382b894"
format_stamp: "Formatted at 2026-09-23 18:11:21 on dist-test-slave-wsx5"
server_key: "8013ae1bdc4c27a020db167067281a20"
server_key_iv: "255e09945d1b539f5eee517befd368a4"
server_key_version: "encryptionkey@0"
I20260923 18:11:21.834200 3765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:21.850402 3765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:21.853560 3765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:21.853749 3765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:21.854388 3765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:21.855221 3765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:21.855268 3765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:21.855321 3765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:21.855346 3765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:21.864392 3765 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36237
I20260923 18:11:21.864533 3893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36237 every 8 connection(s)
I20260923 18:11:21.865537 3765 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:21.870345 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 3765
I20260923 18:11:21.870481 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:11:21.870781 27951 external_mini_cluster.cc:1428] Setting key aa398431f6660d8a0af13c5a4d02300a
Sep 23 18:11:21 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187081, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:11:21.881594 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:21.867561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:51295 (local address 127.27.75.254:34473)
0923 18:11:21.867759 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:21.867764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:21.868702 (+ 938us) server_negotiation.cc:407] Connection header received
0923 18:11:21.870033 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:21.870036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:21.870109 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:21.870184 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:21.871689 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:21.872413 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:21.874255 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:21.874670 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:21.874800 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:21.877743 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:21.877746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:21.877753 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:21.877781 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:21.879115 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:21.879780 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:21.879782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:21.879784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:21.879856 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:21.880250 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:21.880252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:21.880254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:21.880448 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:21.880527 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:21.881236 (+ 709us) server_negotiation.cc:299] Negotiation successful
0923 18:11:21.881360 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 18:11:21.882272 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:21.867969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:51295)
0923 18:11:21.868487 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:21.868581 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:21.869811 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:21.870407 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:21.870417 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:21.870932 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:21.871547 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:21.871563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:21.872623 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:21.872627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:21.874088 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:21.874102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:21.874857 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:21.874860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:21.875017 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:21.875677 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:21.875695 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:21.877528 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:21.879243 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:21.879249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:21.879270 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:21.879675 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:21.879939 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:21.879942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:21.879944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:21.880147 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:21.880557 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:21.880565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:21.880905 (+ 340us) client_negotiation.cc:769] Sending connection context
0923 18:11:21.881083 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 18:11:21.881420 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:11:21.883584 3894 heartbeater.cc:347] Connected to a master server at 127.27.75.254:34473
I20260923 18:11:21.883822 3894 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:21.884321 3894 heartbeater.cc:511] Master 127.27.75.254:34473 requested a full tablet report, sending...
I20260923 18:11:21.885871 3703 ts_manager.cc:194] Registered new tserver with Master: 3fd8de4b2b3140f2998a1f72d382b894 (127.27.75.193:36237)
I20260923 18:11:21.887431 3703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:51295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:11:21.900846 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34473
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:22.012040 3899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:22.012323 3899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:22.012384 3899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:22.016511 3899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:22.016594 3899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:22.016678 3899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:22.022079 3899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:22.023142 3899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:22.024111 3899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:22.033537 3905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:22.033540 3907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:22.033674 3899 server_base.cc:1034] running on GCE node
W20260923 18:11:22.033537 3904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:22.034246 3899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:22.034901 3899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:22.036055 3899 hybrid_clock.cc:648] HybridClock initialized: now 1790187082036038 us; error 32 us; skew 500 ppm
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:22.038890 3899 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:11:22.039846 3899 webserver.cc:460] Webserver started at http://127.27.75.194:38899/ using document root <none> and password file <none>
I20260923 18:11:22.040359 3899 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:22.040416 3899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:22.040599 3899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:22.042694 3899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "e4c1d81ccdec4db888de43467c064226"
format_stamp: "Formatted at 2026-09-23 18:11:22 on dist-test-slave-wsx5"
server_key: "d20c3aa1c9400316547c490b4436538e"
server_key_iv: "f6a1528b594624c625aa78b1146f9f7c"
server_key_version: "encryptionkey@0"
I20260923 18:11:22.043257 3899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "e4c1d81ccdec4db888de43467c064226"
format_stamp: "Formatted at 2026-09-23 18:11:22 on dist-test-slave-wsx5"
server_key: "d20c3aa1c9400316547c490b4436538e"
server_key_iv: "f6a1528b594624c625aa78b1146f9f7c"
server_key_version: "encryptionkey@0"
I20260923 18:11:22.047164 3899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:11:22.049643 3914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:22.050880 3899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:11:22.051000 3899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "e4c1d81ccdec4db888de43467c064226"
format_stamp: "Formatted at 2026-09-23 18:11:22 on dist-test-slave-wsx5"
server_key: "d20c3aa1c9400316547c490b4436538e"
server_key_iv: "f6a1528b594624c625aa78b1146f9f7c"
server_key_version: "encryptionkey@0"
I20260923 18:11:22.051107 3899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:22.087541 3899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:22.090830 3899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:22.091017 3899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:22.091624 3899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:22.092442 3899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:22.092490 3899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:22.092542 3899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:22.092571 3899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:22.101573 3899 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:40555
I20260923 18:11:22.101600 4027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:40555 every 8 connection(s)
I20260923 18:11:22.102496 3899 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:22.107945 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 3899
I20260923 18:11:22.108064 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:11:22.108322 27951 external_mini_cluster.cc:1428] Setting key f826108be36a293c7e5663216e1c79a4
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:11:22.118221 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:22.104529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:45285 (local address 127.27.75.254:34473)
0923 18:11:22.104683 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:22.104686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:22.105586 (+ 900us) server_negotiation.cc:407] Connection header received
0923 18:11:22.106965 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:22.106969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:22.107029 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:22.107102 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:22.108571 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.109318 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.111034 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.111275 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.111367 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:22.114097 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:22.114101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:22.114104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:22.114138 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:22.115793 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:22.116420 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:22.116425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:22.116427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:22.116529 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:22.116942 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:22.116946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:22.116948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:22.117255 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:22.117365 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:22.117934 (+ 569us) server_negotiation.cc:299] Negotiation successful
0923 18:11:22.118079 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 18:11:22.119036 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:22.104901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:45285)
0923 18:11:22.105426 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:22.105473 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:22.106737 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:22.107325 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:22.107332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:22.107712 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:22.108325 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:22.108352 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.109488 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.109491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:22.110885 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:22.110898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.111430 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.111434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:22.111656 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:22.112258 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:22.112277 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:22.113867 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:22.115948 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:22.115953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:22.115969 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:22.116287 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:22.116677 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:22.116680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:22.116681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:22.116842 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:22.117363 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:22.117369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:22.117673 (+ 304us) client_negotiation.cc:769] Sending connection context
0923 18:11:22.117871 (+ 198us) client_negotiation.cc:240] Negotiation successful
0923 18:11:22.118164 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":152,"threads_started":1}
I20260923 18:11:22.120321 4028 heartbeater.cc:347] Connected to a master server at 127.27.75.254:34473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
I20260923 18:11:22.120574 4028 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:22.121083 4028 heartbeater.cc:511] Master 127.27.75.254:34473 requested a full tablet report, sending...
I20260923 18:11:22.122169 3703 ts_manager.cc:194] Registered new tserver with Master: e4c1d81ccdec4db888de43467c064226 (127.27.75.194:40555)
I20260923 18:11:22.122728 3703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:45285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:11:22.137233 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34473
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:22.247220 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:22.247486 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:22.247531 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:22.251528 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:22.251600 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:22.251660 4033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:22.257004 4033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:22.258080 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:22.259083 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:22.268674 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:22.268752 4033 server_base.cc:1034] running on GCE node
W20260923 18:11:22.268668 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:22.268679 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:22.269447 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:22.270105 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:22.271286 4033 hybrid_clock.cc:648] HybridClock initialized: now 1790187082271266 us; error 35 us; skew 500 ppm
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:22.274132 4033 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:11:22.275002 4033 webserver.cc:460] Webserver started at http://127.27.75.195:33649/ using document root <none> and password file <none>
I20260923 18:11:22.275527 4033 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:22.275593 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:22.275771 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:22.277806 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "dc9296f9ff4544219478fdf4f2487b7a"
format_stamp: "Formatted at 2026-09-23 18:11:22 on dist-test-slave-wsx5"
server_key: "82dd16a5523c9071abc53e6a34332fa5"
server_key_iv: "d08ba4e3c87adcfb2aaba764cf0b15e1"
server_key_version: "encryptionkey@0"
I20260923 18:11:22.278318 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "dc9296f9ff4544219478fdf4f2487b7a"
format_stamp: "Formatted at 2026-09-23 18:11:22 on dist-test-slave-wsx5"
server_key: "82dd16a5523c9071abc53e6a34332fa5"
server_key_iv: "d08ba4e3c87adcfb2aaba764cf0b15e1"
server_key_version: "encryptionkey@0"
I20260923 18:11:22.282078 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:11:22.284515 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:22.285787 4033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:11:22.285883 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "dc9296f9ff4544219478fdf4f2487b7a"
format_stamp: "Formatted at 2026-09-23 18:11:22 on dist-test-slave-wsx5"
server_key: "82dd16a5523c9071abc53e6a34332fa5"
server_key_iv: "d08ba4e3c87adcfb2aaba764cf0b15e1"
server_key_version: "encryptionkey@0"
I20260923 18:11:22.285989 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:22.319129 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:22.322845 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:22.323024 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:22.323666 4033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:22.324541 4033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:22.324589 4033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:22.324631 4033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:22.324645 4033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:22.334033 4033 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:43155
I20260923 18:11:22.334066 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:43155 every 8 connection(s)
I20260923 18:11:22.334968 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:22.344271 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 4033
I20260923 18:11:22.344398 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:11:22.344658 27951 external_mini_cluster.cc:1428] Setting key a8f73c8f7816ba5b81ef14401e19058f
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:22.349833 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:22.336800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:44085 (local address 127.27.75.254:34473)
0923 18:11:22.336935 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:22.336939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:22.338070 (+ 1131us) server_negotiation.cc:407] Connection header received
0923 18:11:22.339644 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:22.339647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:22.339702 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:22.339783 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:22.341176 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.341742 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.343019 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.343259 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.343367 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:22.346313 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:22.346317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:22.346320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:22.346350 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:22.347728 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:22.348256 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:22.348259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:22.348260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:22.348340 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:22.348746 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:22.348749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:22.348750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:22.348949 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:22.349024 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:22.349588 (+ 564us) server_negotiation.cc:299] Negotiation successful
0923 18:11:22.349702 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 18:11:22.350756 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:22.337222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:44085)
0923 18:11:22.337832 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:22.337896 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:22.339340 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:22.339961 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:22.339971 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:22.340379 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:22.340951 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:22.340966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.341892 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.341895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:22.342870 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:22.342881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.343427 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.343430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:22.343627 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:22.344321 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:22.344351 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:22.346115 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:22.347869 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:22.347875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:22.347888 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:22.348158 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:22.348507 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:22.348510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:22.348511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:22.348657 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:22.349028 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:22.349034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:22.349345 (+ 311us) client_negotiation.cc:769] Sending connection context
0923 18:11:22.349513 (+ 168us) client_negotiation.cc:240] Negotiation successful
0923 18:11:22.349897 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":134,"threads_started":1}
I20260923 18:11:22.351868 4162 heartbeater.cc:347] Connected to a master server at 127.27.75.254:34473
I20260923 18:11:22.352111 4162 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:22.352581 4162 heartbeater.cc:511] Master 127.27.75.254:34473 requested a full tablet report, sending...
I20260923 18:11:22.353626 3703 ts_manager.cc:194] Registered new tserver with Master: dc9296f9ff4544219478fdf4f2487b7a (127.27.75.195:43155)
I20260923 18:11:22.354195 3703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:44085
I20260923 18:11:22.359232 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab.
Entry for principal oryx/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab.
WARNING: no policy specified for oryx/127.27.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab.
Entry for principal oryx/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab.
WARNING: no policy specified for oryx/127.27.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab.
Entry for principal oryx/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab.
I20260923 18:11:22.433024 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 3765
I20260923 18:11:22.438230 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 3899
I20260923 18:11:22.444373 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 4033
I20260923 18:11:22.448580 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 3671
I20260923 18:11:22.453804 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34473
--webserver_interface=127.27.75.254
--webserver_port=37011
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:22.564893 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:22.565229 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:22.565285 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:22.569453 4171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:22.569522 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:22.569542 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:22.569557 4171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:22.569571 4171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:22.575194 4171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:34473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:22.576316 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:22.577409 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:22.586272 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:22.586324 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:22.586381 4171 server_base.cc:1034] running on GCE node
W20260923 18:11:22.586328 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:22.587150 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:22.588299 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:22.589478 4171 hybrid_clock.cc:648] HybridClock initialized: now 1790187082589461 us; error 36 us; skew 500 ppm
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:22.592376 4171 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:11:22.593313 4171 webserver.cc:460] Webserver started at http://127.27.75.254:37011/ using document root <none> and password file <none>
I20260923 18:11:22.593904 4171 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:22.593986 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:22.597857 4171 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:11:22.600181 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:22.601619 4171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:22.601745 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "4a49256b7f4c4821aa9f88d621b51442"
format_stamp: "Formatted at 2026-09-23 18:11:21 on dist-test-slave-wsx5"
server_key: "9468ac2270ca9bc72145ef53dba14ee4"
server_key_iv: "fe4cc001ae2d11c2b9a7012245279f51"
server_key_version: "encryptionkey@0"
I20260923 18:11:22.602304 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:22.616535 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:22.619586 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:22.619768 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:22.626793 4171 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:34473
I20260923 18:11:22.626807 4238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:34473 every 8 connection(s)
I20260923 18:11:22.627888 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:22.629685 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 4171
I20260923 18:11:22.630101 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:36237
--local_ip_for_outbound_sockets=127.27.75.193
--tserver_master_addrs=127.27.75.254:34473
--webserver_port=45029
--webserver_interface=127.27.75.193
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.75.193
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:11:22.632653 4239 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:11:22.634932 4239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442: Bootstrap starting.
I20260923 18:11:22.656940 4239 log.cc:824] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:22.665436 4239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:11:22.665854 4239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442: Bootstrap complete.
I20260923 18:11:22.671231 4239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } }
I20260923 18:11:22.671819 4239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a49256b7f4c4821aa9f88d621b51442, State: Initialized, Role: FOLLOWER
I20260923 18:11:22.672427 4239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } }
I20260923 18:11:22.672600 4239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:22.672672 4239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:22.672773 4239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:11:22.675465 4239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } }
I20260923 18:11:22.675860 4239 leader_election.cc:302] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a49256b7f4c4821aa9f88d621b51442; no voters:
I20260923 18:11:22.676239 4239 leader_election.cc:288] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 18:11:22.676374 4243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 18:11:22.676707 4243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [term 2 LEADER]: Becoming Leader. State: Replica: 4a49256b7f4c4821aa9f88d621b51442, State: Running, Role: LEADER
I20260923 18:11:22.677033 4243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } }
I20260923 18:11:22.677627 4239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:22.678550 4244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4a49256b7f4c4821aa9f88d621b51442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } } }
I20260923 18:11:22.678519 4245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a49256b7f4c4821aa9f88d621b51442. Latest consensus state: current_term: 2 leader_uuid: "4a49256b7f4c4821aa9f88d621b51442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a49256b7f4c4821aa9f88d621b51442" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34473 } } }
I20260923 18:11:22.678658 4244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:22.678660 4245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:22.679036 4252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:22.682276 4252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:22.683188 4252 catalog_manager.cc:1259] Loaded cluster ID: 7f777d0ef7874189bf92a1f094152424
I20260923 18:11:22.683257 4252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:22.685750 4252 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:22.686546 4252 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a49256b7f4c4821aa9f88d621b51442: Loaded TSK: 0
I20260923 18:11:22.687844 4252 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:11:22.759946 4241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:22.760228 4241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:22.760273 4241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:22.764470 4241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:22.764554 4241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:22.764616 4241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:22.770555 4241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.193.keytab
--principal=oryx/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:36237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=45029
--tserver_master_addrs=127.27.75.254:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:22.771649 4241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:22.772696 4241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:22.782475 4266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:22.782485 4268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:22.782483 4265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:22.782974 4241 server_base.cc:1034] running on GCE node
I20260923 18:11:22.783449 4241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:22.784271 4241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:22.785470 4241 hybrid_clock.cc:648] HybridClock initialized: now 1790187082785413 us; error 65 us; skew 500 ppm
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187082, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:22.788930 4241 init.cc:377] Logged in from keytab as oryx/127.27.75.193@KRBTEST.COM (short username oryx)
I20260923 18:11:22.789940 4241 webserver.cc:460] Webserver started at http://127.27.75.193:45029/ using document root <none> and password file <none>
I20260923 18:11:22.790480 4241 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:22.790555 4241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:22.794404 4241 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:11:22.796362 4275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:22.797616 4241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:11:22.797717 4241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "3fd8de4b2b3140f2998a1f72d382b894"
format_stamp: "Formatted at 2026-09-23 18:11:21 on dist-test-slave-wsx5"
server_key: "8013ae1bdc4c27a020db167067281a20"
server_key_iv: "255e09945d1b539f5eee517befd368a4"
server_key_version: "encryptionkey@0"
I20260923 18:11:22.798250 4241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:22.812664 4241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:22.816035 4241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:22.816222 4241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:22.816831 4241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:22.817723 4241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:22.817770 4241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:22.817812 4241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:22.817842 4241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:22.825966 4241 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36237
I20260923 18:11:22.826009 4389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36237 every 8 connection(s)
I20260923 18:11:22.826917 4241 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:22.827194 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 4241
I20260923 18:11:22.827653 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:40555
--local_ip_for_outbound_sockets=127.27.75.194
--tserver_master_addrs=127.27.75.254:34473
--webserver_port=38899
--webserver_interface=127.27.75.194
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.75.194
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:11:22.841931 4393 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:22.828937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:43455 (local address 127.27.75.254:34473)
0923 18:11:22.829456 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:22.829476 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:22.829820 (+ 344us) server_negotiation.cc:407] Connection header received
0923 18:11:22.831535 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:22.831551 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:22.831901 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:22.832235 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:22.833887 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.834933 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.836670 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.837250 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.837389 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:22.841255 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:43455: BlockingRecv error: recv got EOF from 127.27.75.193:43455 (error 108)
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":132,"threads_started":1}
I20260923 18:11:22.841977 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:22.829164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:43455)
0923 18:11:22.829648 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:22.829700 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:22.831061 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:22.832397 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:22.832407 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:22.832917 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:22.833679 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:22.833701 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.835099 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.835103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:22.836524 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:22.836537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.837422 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.837427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:22.837625 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:22.838223 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:22.838246 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:22.840989 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":130,"threads_started":1}
W20260923 18:11:22.842422 4390 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:34473 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.75.254:34473: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:22.847702 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:22.842891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:60339)
0923 18:11:22.843003 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:22.843018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:22.843101 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:22.843342 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:22.843345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:22.843508 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:22.843609 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:22.843615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.844347 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.844351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:22.844904 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:22.844912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.845380 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.845384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:22.845477 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:22.845908 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:22.845919 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:22.847553 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 18:11:22.847815 4393 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:22.842969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:60339 (local address 127.27.75.254:34473)
0923 18:11:22.843097 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:22.843101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:22.843114 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:22.843202 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:22.843204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:22.843247 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:22.843312 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:22.843729 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.844246 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.845014 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.845274 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.845367 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:22.847705 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:60339: BlockingRecv error: recv got EOF from 127.27.75.193:60339 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:22.852775 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:22.848339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:38167)
0923 18:11:22.848435 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:22.848450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:22.848522 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:22.848732 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:22.848735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:22.848868 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:22.848939 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:22.848944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.849701 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.849704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:22.850216 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:22.850223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.850633 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.850637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:22.850710 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:22.851126 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:22.851139 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:22.852644 (+ 1505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 18:11:22.852871 4393 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:22.848430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:38167 (local address 127.27.75.254:34473)
0923 18:11:22.848532 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:22.848536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:22.848549 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:22.848588 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:22.848590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:22.848632 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:22.848704 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:22.849040 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.849600 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.850318 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:22.850531 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:22.850612 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:22.852766 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:38167: BlockingRecv error: recv got EOF from 127.27.75.193:38167 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260923 18:11:22.853278 4390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:11:22.943672 4392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:22.943948 4392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:22.943996 4392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:22.948112 4392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:22.948184 4392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:22.948246 4392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:22.953637 4392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.194.keytab
--principal=oryx/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:40555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=38899
--tserver_master_addrs=127.27.75.254:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:22.954638 4392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:22.955658 4392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:22.965269 4400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:22.965265 4402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:22.965270 4399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:22.965351 4392 server_base.cc:1034] running on GCE node
I20260923 18:11:22.966020 4392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:22.966689 4392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:22.967876 4392 hybrid_clock.cc:648] HybridClock initialized: now 1790187082967855 us; error 38 us; skew 500 ppm
Sep 23 18:11:22 dist-test-slave-wsx5 krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187082, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:22.970743 4392 init.cc:377] Logged in from keytab as oryx/127.27.75.194@KRBTEST.COM (short username oryx)
I20260923 18:11:22.971709 4392 webserver.cc:460] Webserver started at http://127.27.75.194:38899/ using document root <none> and password file <none>
I20260923 18:11:22.972258 4392 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:22.972350 4392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:22.976514 4392 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:11:22.978972 4409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:22.980458 4392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:11:22.980587 4392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "e4c1d81ccdec4db888de43467c064226"
format_stamp: "Formatted at 2026-09-23 18:11:22 on dist-test-slave-wsx5"
server_key: "d20c3aa1c9400316547c490b4436538e"
server_key_iv: "f6a1528b594624c625aa78b1146f9f7c"
server_key_version: "encryptionkey@0"
I20260923 18:11:22.981211 4392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:22.999919 4392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:23.003242 4392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:23.003420 4392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:23.004027 4392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:23.004864 4392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:23.004910 4392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:23.004966 4392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:23.004993 4392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:23.013859 4392 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:40555
I20260923 18:11:23.013907 4522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:40555 every 8 connection(s)
I20260923 18:11:23.014904 4392 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:23.016038 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 4392
I20260923 18:11:23.016501 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:43155
--local_ip_for_outbound_sockets=127.27.75.195
--tserver_master_addrs=127.27.75.254:34473
--webserver_port=33649
--webserver_interface=127.27.75.195
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.75.195
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:11:23.031597 4393 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:23.016965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:55253 (local address 127.27.75.254:34473)
0923 18:11:23.017143 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:23.017148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:23.020677 (+ 3529us) server_negotiation.cc:407] Connection header received
0923 18:11:23.022592 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:23.022596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:23.022663 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:23.022746 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:23.024632 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.025252 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.027097 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.027344 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.027439 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.031431 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:55253: BlockingRecv error: recv got EOF from 127.27.75.194:55253 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 18:11:23.032294 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:23.017400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:55253)
0923 18:11:23.020391 (+ 2991us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:23.020470 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:23.022275 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:23.023005 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:23.023016 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:23.023573 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:23.024350 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.024372 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.025391 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.025395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.026912 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.026925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.027512 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.027516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.027731 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.028347 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:23.028368 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:23.031258 (+ 2890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2713,"thread_start_us":126,"threads_started":1}
W20260923 18:11:23.032840 4523 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:34473 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.75.254:34473: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:23.038583 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:23.033404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:52037)
0923 18:11:23.033607 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:23.033622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:23.033711 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:23.033932 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:23.033934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:23.034100 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:23.034199 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.034204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.034944 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.034947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.035496 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.035505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.035928 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.035932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.036018 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.036479 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:23.036491 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:23.038441 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116}
W20260923 18:11:23.038695 4393 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:23.033535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:52037 (local address 127.27.75.254:34473)
0923 18:11:23.033669 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:23.033674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:23.033688 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:23.033795 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:23.033798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:23.033839 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:23.033901 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:23.034308 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.034837 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.035608 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.035828 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.035916 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.038571 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:52037: BlockingRecv error: recv got EOF from 127.27.75.194:52037 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:23.044752 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:23.039205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:52865)
0923 18:11:23.039363 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:23.039375 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:23.039456 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:23.039668 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:23.039670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:23.039828 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:23.039921 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.039927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.040616 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.040619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.041451 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.041463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.041923 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.041927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.042005 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.042475 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:23.042490 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:23.044602 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260923 18:11:23.045034 4523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:11:23.044919 4393 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:23.039266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:52865 (local address 127.27.75.254:34473)
0923 18:11:23.039366 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:23.039370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:23.039414 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:11:23.039533 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:23.039534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:23.039570 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:23.039633 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:23.040021 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.040522 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.041587 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.041814 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.041904 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.044814 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:52865: BlockingRecv error: recv got EOF from 127.27.75.194:52865 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260923 18:11:23.132105 4525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:23.132392 4525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:23.132457 4525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:23.136615 4525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:23.136687 4525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:23.136770 4525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:23.142129 4525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/krb5kdc/oryx_127.27.75.195.keytab
--principal=oryx/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:43155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=33649
--tserver_master_addrs=127.27.75.254:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:23.143159 4525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:23.144176 4525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:23.153947 4534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:23.153939 4532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:23.153939 4531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:23.153991 4525 server_base.cc:1034] running on GCE node
I20260923 18:11:23.154627 4525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:23.155277 4525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:23.156458 4525 hybrid_clock.cc:648] HybridClock initialized: now 1790187083156425 us; error 47 us; skew 500 ppm
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187083, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:23.159530 4525 init.cc:377] Logged in from keytab as oryx/127.27.75.195@KRBTEST.COM (short username oryx)
I20260923 18:11:23.160459 4525 webserver.cc:460] Webserver started at http://127.27.75.195:33649/ using document root <none> and password file <none>
I20260923 18:11:23.161038 4525 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:23.161149 4525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:23.165295 4525 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260923 18:11:23.167666 4541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:23.169106 4525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:11:23.169284 4525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "dc9296f9ff4544219478fdf4f2487b7a"
format_stamp: "Formatted at 2026-09-23 18:11:22 on dist-test-slave-wsx5"
server_key: "82dd16a5523c9071abc53e6a34332fa5"
server_key_iv: "d08ba4e3c87adcfb2aaba764cf0b15e1"
server_key_version: "encryptionkey@0"
I20260923 18:11:23.169880 4525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:23.184356 4525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:23.187697 4525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:23.187880 4525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:23.188469 4525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:23.189374 4525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:23.189426 4525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:23.189482 4525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:23.189508 4525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:23.198172 4525 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:43155
I20260923 18:11:23.198251 4654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:43155 every 8 connection(s)
I20260923 18:11:23.199240 4525 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:23.203284 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 4525
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 18:11:23.212229 4393 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:23.201220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:53197 (local address 127.27.75.254:34473)
0923 18:11:23.201372 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:23.201377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:23.202330 (+ 953us) server_negotiation.cc:407] Connection header received
0923 18:11:23.203779 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:23.203783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:23.203846 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:23.203937 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:23.205492 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.206192 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.207775 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.208222 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.208355 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.212102 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:53197: BlockingRecv error: recv got EOF from 127.27.75.195:53197 (error 108)
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":52,"spinlock_wait_cycles":3200}
I20260923 18:11:23.212965 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:23.201676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:53197)
0923 18:11:23.202177 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:23.202220 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:23.203541 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:23.204122 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:23.204132 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:23.204653 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:23.205345 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.205361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.206329 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.206331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.207672 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.207684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.208375 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.208379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.208576 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.209114 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:23.209155 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:23.211927 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":153,"threads_started":1}
W20260923 18:11:23.213477 4655 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:34473 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.75.254:34473: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
I20260923 18:11:23.214702 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:23.204526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.27.75.254:34473)
0923 18:11:23.204827 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:23.204831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:23.204845 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:23.204883 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:23.204886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:23.204927 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:23.205016 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:23.205957 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.206516 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.207726 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.208222 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.208355 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.210099 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:23.210107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:23.210139 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:23.210178 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:23.211981 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:23.212530 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:23.212536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:23.212541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:23.212671 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:23.212993 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:23.212996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:23.212997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:23.213502 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:23.213695 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:23.214310 (+ 615us) server_negotiation.cc:299] Negotiation successful
0923 18:11:23.214569 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":217,"spinlock_wait_cycles":7808,"thread_start_us":84,"threads_started":1}
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:23.219480 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:23.213943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:54039)
0923 18:11:23.214056 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:23.214071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:23.214147 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:23.214401 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:23.214405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:23.214555 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:23.214659 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.214665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.215415 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.215418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.216172 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.216183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.216789 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.216793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.216893 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.217451 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:23.217466 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:23.219370 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 18:11:23.219655 4393 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:23.214041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:54039 (local address 127.27.75.254:34473)
0923 18:11:23.214173 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:23.214178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:23.214189 (+ 11us) server_negotiation.cc:407] Connection header received
0923 18:11:23.214220 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:23.214222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:23.214286 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:23.214375 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:23.214770 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.215305 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.216309 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.216645 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.216748 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.219538 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:54039: BlockingRecv error: recv got EOF from 127.27.75.195:54039 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:23.224840 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:23.220103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:60787)
0923 18:11:23.220201 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:23.220213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:23.220286 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:23.220538 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:23.220540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:23.220727 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:23.220806 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.220812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.221573 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.221577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.222113 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.222120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.222519 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.222523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.222606 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.223025 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:23.223037 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:23.224726 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 18:11:23.225013 4393 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:23.220198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:60787 (local address 127.27.75.254:34473)
0923 18:11:23.220360 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:23.220364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:23.220376 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:11:23.220409 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:23.220411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:23.220446 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:23.220508 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:23.220912 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.221477 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.222217 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.222427 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.222508 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.224891 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:60787: BlockingRecv error: recv got EOF from 127.27.75.195:60787 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260923 18:11:23.225083 4655 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:23 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:23.859846 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:23.854051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:43007)
0923 18:11:23.854345 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:23.854359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:23.854461 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:23.854679 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:23.854682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:23.854859 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:23.854970 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.854976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.855724 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.855726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.856339 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:23.856348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.856867 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.856871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:23.856957 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.857564 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:23.857576 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:23.859666 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":91,"threads_started":1}
W20260923 18:11:23.859975 4661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:23.854141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:43007 (local address 127.27.75.254:34473)
0923 18:11:23.854425 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:23.854433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:23.854447 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:23.854530 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:23.854532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:23.854573 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:23.854649 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:23.855068 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.855607 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.856476 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:23.856731 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:23.856822 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:23.859847 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:43007: BlockingRecv error: recv got EOF from 127.27.75.193:43007 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":112,"threads_started":1}
Sep 23 18:11:24 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:24 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:24.051014 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:24.045757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:49317)
0923 18:11:24.046012 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:24.046025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:24.046131 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:24.046409 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:24.046411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:24.046622 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:24.046722 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:24.046728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.047540 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.047543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:24.048193 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:24.048203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.048640 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.048642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:24.048701 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:24.049202 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:24.049214 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:24.050891 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
W20260923 18:11:24.051193 4661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:24.045892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:49317 (local address 127.27.75.254:34473)
0923 18:11:24.046101 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:24.046104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:24.046121 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:11:24.046234 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:24.046236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:24.046279 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:24.046351 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:24.046832 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.047437 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.048300 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.048514 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.048634 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:24.051076 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:49317: BlockingRecv error: recv got EOF from 127.27.75.194:49317 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 23 18:11:24 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:24 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:24.231510 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:24.225865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:40687)
0923 18:11:24.226158 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:24.226171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:24.226274 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:24.226592 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:24.226594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:24.226766 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:24.226899 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:24.226906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.227874 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.227877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:24.228581 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:24.228593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.229001 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.229003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:24.229093 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:24.229554 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:24.229565 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:24.231387 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":73,"threads_started":1}
W20260923 18:11:24.231642 4661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:24.225969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:40687 (local address 127.27.75.254:34473)
0923 18:11:24.226115 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:24.226118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:24.226235 (+ 117us) server_negotiation.cc:407] Connection header received
0923 18:11:24.226398 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:24.226401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:24.226456 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:24.226575 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:24.227044 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.227764 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.228694 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.228913 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.228993 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:24.231530 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:40687: BlockingRecv error: recv got EOF from 127.27.75.195:40687 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 23 18:11:24 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:24 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:24.866475 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:24.860795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:32921)
0923 18:11:24.861073 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:24.861089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:24.861226 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:24.861464 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:24.861465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:24.861673 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:24.861789 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:24.861796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.862612 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.862617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:24.863210 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:24.863221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.863715 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.863718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:24.863799 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:24.864267 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:24.864279 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:24.866346 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
W20260923 18:11:24.866667 4665 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:24.860912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:32921 (local address 127.27.75.254:34473)
0923 18:11:24.861252 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:24.861256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:24.861270 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:24.861311 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:24.861313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:24.861355 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:24.861437 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:24.861893 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.862450 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.863345 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:24.863596 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:24.863692 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:24.866532 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:32921: BlockingRecv error: recv got EOF from 127.27.75.193:32921 (error 108)
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":101,"threads_started":1}
Sep 23 18:11:25 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:25 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:25.057288 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:25.051787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:54349)
0923 18:11:25.052047 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:25.052061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:25.052180 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:25.052411 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:25.052413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:25.052642 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:25.052746 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:25.052752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.053548 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.053551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:25.054121 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:25.054129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.054598 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.054602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:25.054676 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:25.055160 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:25.055172 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:25.057090 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
W20260923 18:11:25.057391 4665 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:25.051894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:54349 (local address 127.27.75.254:34473)
0923 18:11:25.052091 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:25.052095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:25.052148 (+ 53us) server_negotiation.cc:407] Connection header received
0923 18:11:25.052257 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:25.052259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:25.052302 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:25.052369 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:25.052847 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.053420 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.054236 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.054456 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.054539 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:25.057288 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:54349: BlockingRecv error: recv got EOF from 127.27.75.194:54349 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 23 18:11:25 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:25 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:25.237854 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:25.232380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:51865)
0923 18:11:25.232684 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:25.232698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:25.232783 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:25.233011 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:25.233013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:25.233218 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:25.233328 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:25.233335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.234129 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.234133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:25.234761 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:25.234773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.235264 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.235267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:25.235340 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:25.235829 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:25.235845 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:25.237703 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":95,"threads_started":1}
W20260923 18:11:25.238030 4665 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:25.232561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:51865 (local address 127.27.75.254:34473)
0923 18:11:25.232718 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:25.232721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:25.232739 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:11:25.232866 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:25.232868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:25.232914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:25.232986 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:25.233461 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.234000 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.234894 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.235132 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.235216 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:25.237890 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:51865: BlockingRecv error: recv got EOF from 127.27.75.195:51865 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 23 18:11:25 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:25 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:25.873420 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:25.867526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:50243)
0923 18:11:25.867877 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:25.867895 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:25.867992 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:25.868258 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:25.868260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:25.868434 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:25.868589 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:25.868596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.869457 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.869461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:25.870043 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:25.870052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.870544 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.870548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:25.870629 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:25.871129 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:25.871142 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:25.873231 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":132,"threads_started":1}
W20260923 18:11:25.873551 4669 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:25.867719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:50243 (local address 127.27.75.254:34473)
0923 18:11:25.867943 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:25.867947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:25.867961 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:25.868077 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:25.868079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:25.868128 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:25.868201 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:25.868737 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.869336 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.870178 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:25.870411 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:25.870495 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:25.873379 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:50243: BlockingRecv error: recv got EOF from 127.27.75.193:50243 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
Sep 23 18:11:26 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:26 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:26.063972 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:26.058230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:52705)
0923 18:11:26.058531 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:26.058548 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:26.058661 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:26.058917 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:26.058920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:26.059118 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:26.059213 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:26.059219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.060002 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.060005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:26.060585 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:26.060594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.061041 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.061045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:26.061161 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:26.061710 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:26.061723 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:26.063821 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":104,"threads_started":1}
W20260923 18:11:26.064189 4669 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:26.058425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:52705 (local address 127.27.75.254:34473)
0923 18:11:26.058582 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:26.058586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:26.058604 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:11:26.058760 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:26.058763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:26.058809 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:26.058890 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:26.059332 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.059884 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.060699 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.060926 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.061023 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:26.064055 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:52705: BlockingRecv error: recv got EOF from 127.27.75.194:52705 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 18:11:26 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:26 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:26.244109 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:26.238628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:40019)
0923 18:11:26.238941 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:26.238954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:26.239071 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:26.239328 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:26.239331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:26.239540 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:26.239637 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:26.239642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.240384 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.240387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:26.241008 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:26.241017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.241541 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.241544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:26.241627 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:26.242109 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:26.242127 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:26.243958 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
W20260923 18:11:26.244259 4669 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:26.238711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:40019 (local address 127.27.75.254:34473)
0923 18:11:26.238860 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:26.238865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:26.239051 (+ 186us) server_negotiation.cc:407] Connection header received
0923 18:11:26.239170 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:26.239173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:26.239222 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:26.239289 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:26.239755 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.240276 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.241153 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.241425 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.241515 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:26.244124 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:40019: BlockingRecv error: recv got EOF from 127.27.75.195:40019 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 18:11:26 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:26 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:26.883438 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:26.874241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:57017)
0923 18:11:26.874536 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:26.874551 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:26.874671 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:26.874954 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:26.874957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:26.875165 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:26.875280 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:26.875287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.876125 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.876127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:26.876727 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:26.876738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.877244 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.877247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:26.877322 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:26.881379 (+ 4057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:26.881393 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:26.883296 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
W20260923 18:11:26.883764 4673 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:26.874339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:57017 (local address 127.27.75.254:34473)
0923 18:11:26.874605 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:26.874608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:26.874621 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:26.874755 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:26.874757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:26.874800 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:26.874882 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:26.875382 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.876004 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.876844 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:26.877076 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:26.877209 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:26.883485 (+ 6276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:57017: BlockingRecv error: recv got EOF from 127.27.75.193:57017 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
Sep 23 18:11:27 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:27 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:27.070155 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:27.064835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:48207)
0923 18:11:27.065101 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:27.065113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:27.065244 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:27.065492 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:27.065495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:27.065686 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:27.065776 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:27.065782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.066524 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.066528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:27.067077 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:27.067084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.067527 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.067531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:27.067612 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:27.068121 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:27.068133 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:27.070038 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":105,"threads_started":1}
W20260923 18:11:27.070350 4673 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:27.064885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:48207 (local address 127.27.75.254:34473)
0923 18:11:27.065042 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:27.065046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:27.065204 (+ 158us) server_negotiation.cc:407] Connection header received
0923 18:11:27.065333 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:27.065336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:27.065383 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:27.065450 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:27.065888 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.066416 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.067186 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.067420 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.067510 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:27.070231 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:48207: BlockingRecv error: recv got EOF from 127.27.75.194:48207 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 18:11:27 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:27 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:27.250412 4675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:27.244994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:44681)
0923 18:11:27.245296 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:27.245309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:27.245405 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:27.245661 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:27.245663 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:27.245835 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:27.245932 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:27.245937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.246692 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.246695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:27.247285 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:27.247295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.247792 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.247795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:27.247864 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:27.248387 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:27.248398 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:27.250296 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":130,"threads_started":1}
W20260923 18:11:27.250635 4673 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:27.245087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:44681 (local address 127.27.75.254:34473)
0923 18:11:27.245248 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:27.245252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:27.245363 (+ 111us) server_negotiation.cc:407] Connection header received
0923 18:11:27.245485 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:27.245487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:27.245532 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:27.245630 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:27.246018 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.246533 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.247408 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.247669 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.247749 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:27.250449 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:44681: BlockingRecv error: recv got EOF from 127.27.75.195:44681 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 18:11:27 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:27 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:27.889881 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:27.884373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:33027)
0923 18:11:27.884759 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:27.884775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:27.884895 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:27.885186 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:27.885189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:27.885393 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:27.885489 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:27.885496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.886283 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.886287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:27.886862 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:27.886871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.887319 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.887323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:27.887403 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:27.887897 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:27.887909 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:27.889728 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":132,"threads_started":1}
W20260923 18:11:27.890007 4677 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:27.884471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:33027 (local address 127.27.75.254:34473)
0923 18:11:27.884761 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:27.884765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:27.884852 (+ 87us) server_negotiation.cc:407] Connection header received
0923 18:11:27.884982 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:27.884985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:27.885031 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:27.885115 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:27.885606 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.886173 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.886975 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:27.887210 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:27.887298 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:27.889879 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:33027: BlockingRecv error: recv got EOF from 127.27.75.193:33027 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
Sep 23 18:11:28 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:28 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:28.077001 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:28.071115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:41147)
0923 18:11:28.071495 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:28.071519 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:28.071654 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:28.072014 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:28.072017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:28.072230 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:28.072355 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:28.072364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.073168 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.073172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:28.073915 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:28.073927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.074393 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.074396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:28.074463 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:28.075017 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:28.075029 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:28.076885 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
W20260923 18:11:28.077186 4677 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:28.071294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:41147 (local address 127.27.75.254:34473)
0923 18:11:28.071429 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:28.071433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:28.071630 (+ 197us) server_negotiation.cc:407] Connection header received
0923 18:11:28.071765 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:28.071768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:28.071830 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:28.071914 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:28.072472 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.072995 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.074050 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.074279 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.074369 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:28.077038 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:41147: BlockingRecv error: recv got EOF from 127.27.75.194:41147 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 18:11:28 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:28 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:28.257449 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:28.251465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:38777)
0923 18:11:28.251729 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:28.251742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:28.251853 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:28.252225 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:28.252228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:28.252425 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:28.252561 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:28.252570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.253583 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.253587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:28.254312 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:28.254324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.254816 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.254819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:28.254888 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:28.255420 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:28.255433 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:28.257307 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
W20260923 18:11:28.257619 4677 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:28.251587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:38777 (local address 127.27.75.254:34473)
0923 18:11:28.251722 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:28.251726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:28.251814 (+ 88us) server_negotiation.cc:407] Connection header received
0923 18:11:28.251986 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:28.251989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:28.252048 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:28.252151 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:28.252707 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.253451 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.254446 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.254694 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.254779 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:28.257487 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:38777: BlockingRecv error: recv got EOF from 127.27.75.195:38777 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 23 18:11:28 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:28 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:28.896224 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:28.890738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:51737)
0923 18:11:28.890993 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:28.891009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:28.891111 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:28.891383 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:28.891386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:28.891599 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:28.891699 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:28.891705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.892556 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.892559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:28.893162 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:28.893173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.893722 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.893725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:28.893793 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:28.894269 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:28.894280 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:28.896105 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
W20260923 18:11:28.896363 4681 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:28.890871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:51737 (local address 127.27.75.254:34473)
0923 18:11:28.891134 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:28.891138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:28.891153 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:11:28.891192 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:28.891193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:28.891236 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:28.891313 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:28.891837 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.892403 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.893311 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:28.893604 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:28.893702 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:28.896229 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:51737: BlockingRecv error: recv got EOF from 127.27.75.193:51737 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
Sep 23 18:11:29 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:29 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:29.084110 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:29.077844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:55715)
0923 18:11:29.078224 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:29.078239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:29.078359 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:29.078678 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:29.078681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:29.078957 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:29.079107 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:29.079114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.079988 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.079991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:29.080576 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:29.080585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.081101 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.081105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:29.081257 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:29.081790 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:29.081802 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:29.083967 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":92,"threads_started":1}
W20260923 18:11:29.084328 4681 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:29.077915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:55715 (local address 127.27.75.254:34473)
0923 18:11:29.078065 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:29.078070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:29.078319 (+ 249us) server_negotiation.cc:407] Connection header received
0923 18:11:29.078459 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:29.078463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:29.078519 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:29.078601 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:29.079255 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.079853 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.080716 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.080963 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.081057 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:29.084188 (+ 3131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:55715: BlockingRecv error: recv got EOF from 127.27.75.194:55715 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 18:11:29 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:29 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:29.264060 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:29.258475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:52255)
0923 18:11:29.258835 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:29.258849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:29.258983 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:29.259333 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:29.259337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:29.259542 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:29.259649 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:29.259655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.260418 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.260421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:29.261000 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:29.261009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.261583 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.261587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:29.261665 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:29.262154 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:29.262166 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:29.263899 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":94,"threads_started":1}
W20260923 18:11:29.264161 4681 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:29.258557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:52255 (local address 127.27.75.254:34473)
0923 18:11:29.258718 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:29.258723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:29.258922 (+ 199us) server_negotiation.cc:407] Connection header received
0923 18:11:29.259083 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:29.259087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:29.259146 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:29.259254 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:29.259766 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.260327 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.261148 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.261415 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.261552 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:29.264047 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:52255: BlockingRecv error: recv got EOF from 127.27.75.195:52255 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 18:11:29 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:29 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:29.902979 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:29.897180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:57135)
0923 18:11:29.897574 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:29.897588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:29.897704 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:29.897952 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:29.897955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:29.898142 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:29.898240 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:29.898245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.899061 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.899064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:29.899676 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:29.899687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.900218 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.900221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:29.900295 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:29.900849 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:29.900861 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:29.902855 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":88,"threads_started":1}
W20260923 18:11:29.903191 4685 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:29.897284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:57135 (local address 127.27.75.254:34473)
0923 18:11:29.897530 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:29.897534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:29.897649 (+ 115us) server_negotiation.cc:407] Connection header received
0923 18:11:29.897789 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:29.897791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:29.897837 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:29.897906 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:29.898362 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.898940 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.899817 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:29.900068 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:29.900161 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:29.903048 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:57135: BlockingRecv error: recv got EOF from 127.27.75.193:57135 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":58,"threads_started":1}
Sep 23 18:11:30 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:30 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:30.090392 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:30.084875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:34849)
0923 18:11:30.085243 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:30.085256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:30.085369 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:30.085672 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:30.085674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:30.085924 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:30.086027 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:30.086033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.086779 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.086783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:30.087378 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:30.087386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.087827 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.087831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:30.087911 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:30.088404 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:30.088415 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:30.090239 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":84,"threads_started":1}
W20260923 18:11:30.090528 4685 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:30.084964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:34849 (local address 127.27.75.254:34473)
0923 18:11:30.085180 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:30.085184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:30.085328 (+ 144us) server_negotiation.cc:407] Connection header received
0923 18:11:30.085514 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:30.085516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:30.085562 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:30.085631 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:30.086140 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.086676 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.087500 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.087725 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.087811 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:30.090408 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:34849: BlockingRecv error: recv got EOF from 127.27.75.194:34849 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 23 18:11:30 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:30 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:30.270401 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:30.264853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:60365)
0923 18:11:30.265227 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:30.265244 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:30.265358 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:30.265656 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:30.265659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:30.265848 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:30.265941 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:30.265946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.266691 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.266694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:30.267255 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:30.267263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.267705 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.267709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:30.267787 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:30.268319 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:30.268332 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:30.270245 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":80,"threads_started":1}
W20260923 18:11:30.270514 4685 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:30.264946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:60365 (local address 127.27.75.254:34473)
0923 18:11:30.265155 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:30.265160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:30.265320 (+ 160us) server_negotiation.cc:407] Connection header received
0923 18:11:30.265450 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:30.265453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:30.265542 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:30.265613 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:30.266052 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.266588 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.267371 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.267604 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.267690 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:30.270405 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:60365: BlockingRecv error: recv got EOF from 127.27.75.195:60365 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 23 18:11:30 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:30 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:30.909544 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:30.903770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:53545)
0923 18:11:30.904031 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:30.904044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:30.904135 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:30.904442 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:30.904445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:30.904626 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:30.904729 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:30.904735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.905604 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.905608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:30.906239 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:30.906249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.906777 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.906780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:30.906852 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:30.907352 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:30.907363 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:30.909339 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
W20260923 18:11:30.909662 4689 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:30.903944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:53545 (local address 127.27.75.254:34473)
0923 18:11:30.904182 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:30.904185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:30.904198 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:30.904238 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:30.904240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:30.904282 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:30.904367 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:30.904838 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.905466 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.906366 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:30.906660 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:30.906749 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:30.909530 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:53545: BlockingRecv error: recv got EOF from 127.27.75.193:53545 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
Sep 23 18:11:31 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:31 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:31.097091 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:31.091266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:46947)
0923 18:11:31.091509 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:31.091522 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:31.091623 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:31.091832 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:31.091834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:31.092001 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:31.092097 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:31.092102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.093001 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.093004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:31.093708 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:31.093720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.094226 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.094230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:31.094329 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:31.094991 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:31.095006 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:31.096969 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
W20260923 18:11:31.097301 4689 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:31.091354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:46947 (local address 127.27.75.254:34473)
0923 18:11:31.091508 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:31.091511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:31.091588 (+ 77us) server_negotiation.cc:407] Connection header received
0923 18:11:31.091692 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:31.091694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:31.091736 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:31.091800 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:31.092199 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.092865 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.093849 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.094099 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.094192 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:31.097170 (+ 2978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:46947: BlockingRecv error: recv got EOF from 127.27.75.194:46947 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 23 18:11:31 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:31 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:31.276346 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:31.271263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:60585)
0923 18:11:31.271507 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:31.271521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:31.271613 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:31.271865 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:31.271868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:31.272050 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:31.272149 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:31.272154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.272885 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.272888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:31.273498 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:31.273507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.273966 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.273968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:31.274028 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:31.274519 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:31.274531 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:31.276229 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
W20260923 18:11:31.276484 4689 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:31.271385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:60585 (local address 127.27.75.254:34473)
0923 18:11:31.271563 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:31.271566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:31.271580 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:31.271697 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:31.271699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:31.271742 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:31.271801 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:31.272254 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.272779 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.273635 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.273874 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.273967 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:31.276371 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:60585: BlockingRecv error: recv got EOF from 127.27.75.195:60585 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 23 18:11:31 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:31 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:31.921869 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:31.910431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:53607)
0923 18:11:31.910690 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:31.910704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:31.910794 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:31.911124 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:31.911127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:31.911308 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:31.911411 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:31.911417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.912290 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.912292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:31.912896 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:31.912906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.913515 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.913518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:31.913586 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:31.919567 (+ 5981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:31.919582 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:31.921745 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
W20260923 18:11:31.922098 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:31.910604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:53607 (local address 127.27.75.254:34473)
0923 18:11:31.910837 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:31.910840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:31.910854 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:31.910896 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:31.910898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:31.910938 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:31.911055 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:31.911584 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.912179 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.913008 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:31.913327 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:31.913442 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:31.921922 (+ 8480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:53607: BlockingRecv error: recv got EOF from 127.27.75.193:53607 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
Sep 23 18:11:32 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:32 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:32.103976 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:32.097985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:39287)
0923 18:11:32.098250 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:32.098262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:32.098379 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:32.098658 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:32.098661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:32.098841 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:32.098988 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:32.098994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.099807 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.099810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:32.100397 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:32.100406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.100948 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.100952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:32.101023 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:32.101627 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:32.101641 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:32.103812 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
W20260923 18:11:32.104102 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:32.098087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:39287 (local address 127.27.75.254:34473)
0923 18:11:32.098221 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:32.098224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:32.098332 (+ 108us) server_negotiation.cc:407] Connection header received
0923 18:11:32.098474 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:32.098476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:32.098523 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:32.098602 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:32.099112 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.099684 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.100536 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.100777 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.100890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:32.103993 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:39287: BlockingRecv error: recv got EOF from 127.27.75.194:39287 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 23 18:11:32 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:32 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:32.282852 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:32.277163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:49525)
0923 18:11:32.277459 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:32.277472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:32.277576 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:32.277898 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:32.277900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:32.278097 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:32.278226 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:32.278232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.279127 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.279130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:32.279742 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:32.279751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.280217 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.280221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:32.280286 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:32.280796 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:32.280807 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:32.282692 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
W20260923 18:11:32.282944 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:32.277268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:49525 (local address 127.27.75.254:34473)
0923 18:11:32.277408 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:32.277413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:32.277565 (+ 152us) server_negotiation.cc:407] Connection header received
0923 18:11:32.277747 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:32.277750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:32.277799 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:32.277867 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:32.278340 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.279011 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.279841 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.280082 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.280183 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:32.282835 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:49525: BlockingRecv error: recv got EOF from 127.27.75.195:49525 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 23 18:11:32 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:32 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:32.928208 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:32.922705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:54823)
0923 18:11:32.922951 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:32.922965 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:32.923052 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:32.923302 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:32.923304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:32.923486 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:32.923626 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:32.923632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.924378 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.924381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:32.924987 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:32.924997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.925611 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.925615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:32.925691 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:32.926226 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:32.926239 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:32.928075 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
W20260923 18:11:32.928376 4697 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:32.922795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:54823 (local address 127.27.75.254:34473)
0923 18:11:32.923028 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:32.923031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:32.923045 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:11:32.923137 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:32.923140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:32.923182 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:32.923263 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:32.923731 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.924269 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.925156 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:32.925426 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:32.925529 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:32.928234 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:54823: BlockingRecv error: recv got EOF from 127.27.75.193:54823 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
Sep 23 18:11:33 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:33 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:33.110628 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:33.104922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:60885)
0923 18:11:33.105210 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:33.105227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:33.105334 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:33.105668 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:33.105671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:33.105854 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:33.105959 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:33.105964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.106912 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.106916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:33.107574 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:33.107586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.108043 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.108046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:33.108126 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:33.108588 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:33.108603 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:33.110469 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
W20260923 18:11:33.110744 4697 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:33.105040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:60885 (local address 127.27.75.254:34473)
0923 18:11:33.105210 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:33.105214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:33.105405 (+ 191us) server_negotiation.cc:407] Connection header received
0923 18:11:33.105462 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:33.105465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:33.105514 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:33.105578 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:33.106107 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.106773 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.107698 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.107928 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.108032 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:33.110619 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:60885: BlockingRecv error: recv got EOF from 127.27.75.194:60885 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 23 18:11:33 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:33 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:33.289392 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:33.283672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:54069)
0923 18:11:33.283967 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:33.283981 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:33.284081 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:33.284551 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:33.284554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:33.284735 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:33.284833 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:33.284838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.285803 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.285807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:33.286386 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:33.286395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.286830 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.286832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:33.286890 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:33.287340 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:33.287352 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:33.289273 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
W20260923 18:11:33.289625 4697 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:33.283849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:54069 (local address 127.27.75.254:34473)
0923 18:11:33.283990 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:33.283993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:33.284036 (+ 43us) server_negotiation.cc:407] Connection header received
0923 18:11:33.284151 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:33.284153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:33.284196 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:33.284320 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:33.284935 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.285708 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.286521 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.286742 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.286820 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:33.289450 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:54069: BlockingRecv error: recv got EOF from 127.27.75.195:54069 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 18:11:33 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:33 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:33.935230 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:33.929204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:51285)
0923 18:11:33.929502 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:33.929515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:33.929646 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:33.929930 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:33.929933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:33.930130 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:33.930233 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:33.930239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.931197 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.931201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:33.931861 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:33.931872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.932413 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.932417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:33.932521 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:33.933080 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:33.933094 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:33.935105 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
W20260923 18:11:33.935470 4701 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:33.929335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:51285 (local address 127.27.75.254:34473)
0923 18:11:33.929629 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:33.929633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:33.929648 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:11:33.929729 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:33.929731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:33.929774 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:33.929851 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:33.930398 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.931067 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.931978 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:33.932284 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:33.932391 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:33.935271 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:51285: BlockingRecv error: recv got EOF from 127.27.75.193:51285 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":59,"threads_started":1}
Sep 23 18:11:34 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:34 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:34.116752 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:34.111328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:50695)
0923 18:11:34.111614 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:34.111630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:34.111753 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:34.112006 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:34.112008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:34.112204 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:34.112300 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:34.112307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.113076 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.113080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:34.113706 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:34.113715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.114150 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.114154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:34.114236 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:34.114738 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:34.114751 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:34.116616 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
W20260923 18:11:34.116904 4701 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:34.111433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:50695 (local address 127.27.75.254:34473)
0923 18:11:34.111615 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:34.111620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:34.111703 (+ 83us) server_negotiation.cc:407] Connection header received
0923 18:11:34.111852 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:34.111855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:34.111899 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:34.111965 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:34.112423 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.112968 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.113821 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.114048 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.114135 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:34.116780 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:50695: BlockingRecv error: recv got EOF from 127.27.75.194:50695 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 23 18:11:34 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:34 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:34.295578 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:34.290223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:33821)
0923 18:11:34.290457 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:34.290471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:34.290592 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:34.290885 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:34.290889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:34.291177 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:34.291282 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:34.291289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.292047 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.292051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:34.292619 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:34.292627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.293022 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.293025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:34.293096 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:34.293633 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:34.293644 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:34.295408 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
W20260923 18:11:34.295704 4701 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:34.290273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:33821 (local address 127.27.75.254:34473)
0923 18:11:34.290404 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:34.290408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:34.290571 (+ 163us) server_negotiation.cc:407] Connection header received
0923 18:11:34.290702 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:34.290705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:34.290756 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:34.290827 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:34.291389 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.291937 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.292711 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.292927 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.292997 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:34.295592 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:33821: BlockingRecv error: recv got EOF from 127.27.75.195:33821 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 23 18:11:34 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:34 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:34.942082 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:34.936169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:50221)
0923 18:11:34.936509 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:34.936524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:34.936628 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:34.936872 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:34.936875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:34.937068 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:34.937226 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:34.937234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.938045 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.938048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:34.938682 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:34.938692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.939298 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.939302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:34.939387 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:34.939961 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:34.939972 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:34.941937 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
W20260923 18:11:34.942238 4705 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:34.936317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:50221 (local address 127.27.75.254:34473)
0923 18:11:34.936610 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:34.936613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:34.936630 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:11:34.936702 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:34.936705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:34.936748 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:34.936825 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:34.937370 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.937920 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.938805 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:34.939174 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:34.939263 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:34.942100 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:50221: BlockingRecv error: recv got EOF from 127.27.75.193:50221 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
Sep 23 18:11:35 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:35 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:35.123174 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:35.117583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:50563)
0923 18:11:35.117862 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:35.117876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:35.117977 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:35.118228 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:35.118231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:35.118452 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:35.118578 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:35.118584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.119333 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.119335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:35.119929 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:35.119938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.120385 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.120388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:35.120445 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:35.120973 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:35.120987 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:35.123044 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
W20260923 18:11:35.123365 4705 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:35.117697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:50563 (local address 127.27.75.254:34473)
0923 18:11:35.117845 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:35.117849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:35.117956 (+ 107us) server_negotiation.cc:407] Connection header received
0923 18:11:35.118070 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:35.118072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:35.118122 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:35.118188 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:35.118711 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.119240 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.120062 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.120287 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.120370 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:35.123242 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:50563: BlockingRecv error: recv got EOF from 127.27.75.194:50563 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 18:11:35 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:35 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:35.302523 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:35.296530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:38953)
0923 18:11:35.296813 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:35.296828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:35.296978 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:35.297462 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:35.297464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:35.297681 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:35.297788 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:35.297796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.298565 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.298568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:35.299149 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:35.299157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.299643 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.299647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:35.299717 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:35.300203 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:35.300216 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:35.302396 (+ 2180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
W20260923 18:11:35.302771 4705 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:35.296649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:38953 (local address 127.27.75.254:34473)
0923 18:11:35.296815 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:35.296819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:35.296909 (+ 90us) server_negotiation.cc:407] Connection header received
0923 18:11:35.297083 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:35.297086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:35.297165 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:35.297250 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:35.297908 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.298460 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.299263 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.299487 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.299568 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:35.302564 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:38953: BlockingRecv error: recv got EOF from 127.27.75.195:38953 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 23 18:11:35 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:35 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:35.948350 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:35.942876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:46861)
0923 18:11:35.943150 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:35.943163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:35.943259 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:35.943595 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:35.943597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:35.943776 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:35.943872 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:35.943877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.944732 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.944735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:35.945393 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:35.945406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.945855 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.945858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:35.945949 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:35.946446 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:35.946458 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:35.948230 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
W20260923 18:11:35.948583 4709 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:35.942990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:46861 (local address 127.27.75.254:34473)
0923 18:11:35.943210 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:35.943214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:35.943226 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:11:35.943382 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:35.943384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:35.943483 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:35.943568 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:35.944016 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.944610 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.945516 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:35.945741 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:35.945818 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:35.948386 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:46861: BlockingRecv error: recv got EOF from 127.27.75.193:46861 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":57,"threads_started":1}
Sep 23 18:11:36 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:36 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:36.129632 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:36.123984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:42569)
0923 18:11:36.124247 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:36.124262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:36.124357 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:36.124593 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:36.124595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:36.124770 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:36.124870 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:36.124877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.125837 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.125841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:36.126411 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:36.126420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.126951 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.126954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:36.127020 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:36.127489 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:36.127500 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:36.129466 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
W20260923 18:11:36.129750 4709 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:36.124208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:42569 (local address 127.27.75.254:34473)
0923 18:11:36.124338 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:36.124341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:36.124354 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:11:36.124442 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:36.124444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:36.124488 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:36.124564 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:36.125026 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.125705 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.126550 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.126828 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.126925 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:36.129654 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:42569: BlockingRecv error: recv got EOF from 127.27.75.194:42569 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 23 18:11:36 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:36 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:36.309008 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:36.303329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:56443)
0923 18:11:36.303605 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:36.303618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:36.303723 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:36.303989 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:36.303992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:36.304172 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:36.304269 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:36.304274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.305051 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.305054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:36.305721 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:36.305731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.306174 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.306177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:36.306256 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:36.306803 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:36.306818 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:36.308860 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
W20260923 18:11:36.309104 4709 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:36.303444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:56443 (local address 127.27.75.254:34473)
0923 18:11:36.303611 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:36.303615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:36.303711 (+ 96us) server_negotiation.cc:407] Connection header received
0923 18:11:36.303792 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:36.303794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:36.303846 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:36.303918 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:36.304383 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.304944 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.305822 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.306056 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.306137 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:36.308998 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:56443: BlockingRecv error: recv got EOF from 127.27.75.195:56443 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 18:11:36 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:36 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:36.960204 4712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:36.949243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:53973)
0923 18:11:36.949512 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:36.949527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:36.949624 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:36.950000 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:36.950002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:36.950187 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:36.950289 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:36.950294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.951133 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.951136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:36.951723 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:36.951733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.952283 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.952287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:36.952378 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:36.958209 (+ 5831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:36.958225 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:36.960079 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
W20260923 18:11:36.960547 4713 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:36.949380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:53973 (local address 127.27.75.254:34473)
0923 18:11:36.949703 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:36.949706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:36.949722 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:11:36.949767 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:36.949769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:36.949810 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:36.949904 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:36.950434 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.950993 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.951865 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:36.952146 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:36.952254 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:36.960268 (+ 8014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:53973: BlockingRecv error: recv got EOF from 127.27.75.193:53973 (error 108)
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":85,"threads_started":1}
Sep 23 18:11:37 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:37 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:37.135944 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:37.130481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:50365)
0923 18:11:37.130763 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:37.130779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:37.130895 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:37.131151 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:37.131153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:37.131375 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:37.131489 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:37.131497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.132287 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.132290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:37.132885 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:37.132895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.133349 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.133352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:37.133430 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:37.133938 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:37.133951 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:37.135777 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
W20260923 18:11:37.136018 4713 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:37.130556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:50365 (local address 127.27.75.254:34473)
0923 18:11:37.130769 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:37.130774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:37.130852 (+ 78us) server_negotiation.cc:407] Connection header received
0923 18:11:37.130982 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:37.130985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:37.131032 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:37.131115 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:37.131602 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.132182 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.132999 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.133260 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.133339 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:37.135913 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:50365: BlockingRecv error: recv got EOF from 127.27.75.194:50365 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 23 18:11:37 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:37 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.195@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:37.315526 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:37.309842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.195:47639)
0923 18:11:37.310103 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:37.310118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:37.310235 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:37.310540 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:37.310543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:37.310740 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:37.310835 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:37.310841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.311616 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.311620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:37.312183 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:37.312192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.312632 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.312635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:37.312716 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:37.313439 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:37.313455 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:37.315349 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
W20260923 18:11:37.315666 4713 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:37.309951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:47639 (local address 127.27.75.254:34473)
0923 18:11:37.310082 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:37.310086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:37.310197 (+ 111us) server_negotiation.cc:407] Connection header received
0923 18:11:37.310326 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:37.310329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:37.310378 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:37.310494 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:37.310959 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.311509 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.312299 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.312531 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.312614 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:37.315497 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.195:47639: BlockingRecv error: recv got EOF from 127.27.75.195:47639 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 18:11:37 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:37 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.193@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:37.966657 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:37.961115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.193:55653)
0923 18:11:37.961440 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:37.961457 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:37.961546 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:37.961811 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:37.961814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:37.962022 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:37.962123 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:37.962129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.962910 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.962913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:37.963514 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:37.963524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.964008 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.964012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:37.964085 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:37.964562 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:37.964575 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:37.966534 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":83,"threads_started":1}
W20260923 18:11:37.966840 4717 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:37.961255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:55653 (local address 127.27.75.254:34473)
0923 18:11:37.961532 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:37.961535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:37.961551 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:11:37.961616 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:37.961618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:37.961657 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:37.961738 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:37.962223 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.962784 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.963638 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:37.963872 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:37.963955 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:37.966701 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.193:55653: BlockingRecv error: recv got EOF from 127.27.75.193:55653 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":66,"threads_started":1}
Sep 23 18:11:38 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 18:11:38 dist-test-slave-wsx5 krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.75.194@KRBTEST.COM for oryx/127.27.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 18:11:38.142530 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:38.136816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34473 (local address 127.27.75.194:43797)
0923 18:11:38.137088 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:38.137101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:38.137212 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:38.137564 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:38.137567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:38.137742 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:38.137848 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:38.137854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.138668 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.138671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:38.139293 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:38.139304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.139900 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.139904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:38.139977 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:38.140490 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:38.140503 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:38.142366 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.75.254:34473: Server oryx/127.27.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":71,"threads_started":1}
W20260923 18:11:38.142642 4717 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:38.136886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:43797 (local address 127.27.75.254:34473)
0923 18:11:38.137030 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:38.137035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:38.137279 (+ 244us) server_negotiation.cc:407] Connection header received
0923 18:11:38.137333 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:38.137335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:38.137381 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:38.137497 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:38.137992 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.138553 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.139478 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.139751 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.139884 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:38.142525 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.75.194:43797: BlockingRecv error: recv got EOF from 127.27.75.194:43797 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:11:38.206785 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 4241
I20260923 18:11:38.211915 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 4392
I20260923 18:11:38.216383 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 4525
I20260923 18:11:38.221304 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 4171
2026-09-23T18:11:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17901 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T18:11:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:38Z Disabled control of system clock
I20260923 18:11:38.250058 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:43015
--webserver_interface=127.27.75.254
--webserver_port=0
--builtin_ntp_servers=127.27.75.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:43015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 18:11:38.361533 4722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:38.361821 4722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:38.361856 4722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:38.365866 4722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:38.365929 4722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:38.365942 4722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:38.365958 4722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:38.365972 4722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:38.370780 4722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:43015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:43015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:38.371819 4722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:38.373178 4722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:38.381733 4728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:38.381875 4722 server_base.cc:1034] running on GCE node
W20260923 18:11:38.381767 4730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:38.381767 4727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:38.382599 4722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:38.383723 4722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:38.384889 4722 hybrid_clock.cc:648] HybridClock initialized: now 1790187098384871 us; error 32 us; skew 500 ppm
I20260923 18:11:38.386829 4722 webserver.cc:460] Webserver started at http://127.27.75.254:41169/ using document root <none> and password file <none>
I20260923 18:11:38.387395 4722 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:38.387488 4722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:38.387702 4722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:38.389837 4722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "99b3b0e7441a412eae77c6732d898929"
format_stamp: "Formatted at 2026-09-23 18:11:38 on dist-test-slave-wsx5"
server_key: "7395d57f5e99d0529dcecb358ed92733"
server_key_iv: "b1294ba189ff7759a4d677cd45e56ebc"
server_key_version: "encryptionkey@0"
I20260923 18:11:38.390378 4722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "99b3b0e7441a412eae77c6732d898929"
format_stamp: "Formatted at 2026-09-23 18:11:38 on dist-test-slave-wsx5"
server_key: "7395d57f5e99d0529dcecb358ed92733"
server_key_iv: "b1294ba189ff7759a4d677cd45e56ebc"
server_key_version: "encryptionkey@0"
I20260923 18:11:38.393953 4722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 18:11:38.396342 4736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:38.397670 4722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:11:38.397785 4722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "99b3b0e7441a412eae77c6732d898929"
format_stamp: "Formatted at 2026-09-23 18:11:38 on dist-test-slave-wsx5"
server_key: "7395d57f5e99d0529dcecb358ed92733"
server_key_iv: "b1294ba189ff7759a4d677cd45e56ebc"
server_key_version: "encryptionkey@0"
I20260923 18:11:38.397886 4722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:38.410269 4722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:38.411094 4722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:38.411289 4722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:38.418550 4722 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:43015
I20260923 18:11:38.418566 4788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:43015 every 8 connection(s)
I20260923 18:11:38.419526 4722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:38.422976 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:38.425654 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 4722
I20260923 18:11:38.425760 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:38.426025 27951 external_mini_cluster.cc:1428] Setting key 59bfff5574b3fa78b7e4e11fa4f30d19
I20260923 18:11:38.429484 4789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929: Bootstrap starting.
I20260923 18:11:38.432413 4789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:38.433439 4789 log.cc:824] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:38.436633 4789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929: No bootstrap required, opened a new log
I20260923 18:11:38.438660 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:38.427494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.27.75.254:43015)
0923 18:11:38.428220 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:38.428241 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:38.428286 (+ 45us) server_negotiation.cc:407] Connection header received
0923 18:11:38.429084 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:38.429106 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:38.429749 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:38.430167 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:38.430791 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.432928 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.434753 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.435406 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.435576 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:38.436417 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:38.436424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:38.436442 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:38.436967 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:38.437063 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:38.437093 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:38.437224 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:38.437382 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 18:11:38.437683 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":115,"threads_started":1}
I20260923 18:11:38.441210 4789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b3b0e7441a412eae77c6732d898929" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43015 } }
I20260923 18:11:38.441507 4789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:38.441555 4789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b3b0e7441a412eae77c6732d898929, State: Initialized, Role: FOLLOWER
I20260923 18:11:38.441934 4789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b3b0e7441a412eae77c6732d898929" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43015 } }
I20260923 18:11:38.442049 4789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:38.442093 4789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:38.442163 4789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:38.443120 4789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b3b0e7441a412eae77c6732d898929" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43015 } }
I20260923 18:11:38.443437 4789 leader_election.cc:302] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99b3b0e7441a412eae77c6732d898929; no voters:
I20260923 18:11:38.443730 4789 leader_election.cc:288] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:38.443866 4794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:38.444142 4794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [term 1 LEADER]: Becoming Leader. State: Replica: 99b3b0e7441a412eae77c6732d898929, State: Running, Role: LEADER
I20260923 18:11:38.444466 4794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b3b0e7441a412eae77c6732d898929" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43015 } }
I20260923 18:11:38.444986 4789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:38.446102 4795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99b3b0e7441a412eae77c6732d898929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b3b0e7441a412eae77c6732d898929" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43015 } } }
I20260923 18:11:38.446239 4795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:38.446671 4803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:38.446578 4796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99b3b0e7441a412eae77c6732d898929. Latest consensus state: current_term: 1 leader_uuid: "99b3b0e7441a412eae77c6732d898929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b3b0e7441a412eae77c6732d898929" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43015 } } }
I20260923 18:11:38.447328 4796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:38.449627 4803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:38.455883 4803 catalog_manager.cc:1347] Generated new cluster ID: 86ae9ca743ca403b9e418ffb88d44daf
I20260923 18:11:38.455981 4803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:38.497944 4803 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:38.498915 4803 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:38.504118 4803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99b3b0e7441a412eae77c6732d898929: Generated new TSK 0
I20260923 18:11:38.504791 4803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:11:38.521842 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43015
--builtin_ntp_servers=127.27.75.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 18:11:38.640033 4813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:38.640321 4813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:38.640358 4813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:38.644491 4813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:38.644568 4813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:38.644624 4813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:38.649745 4813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:38.650835 4813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:38.652320 4813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:38.661962 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:38.661962 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:38.661974 4821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:38.662055 4813 server_base.cc:1034] running on GCE node
I20260923 18:11:38.662712 4813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:38.663496 4813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:38.664688 4813 hybrid_clock.cc:648] HybridClock initialized: now 1790187098664660 us; error 39 us; skew 500 ppm
I20260923 18:11:38.666682 4813 webserver.cc:460] Webserver started at http://127.27.75.193:46253/ using document root <none> and password file <none>
I20260923 18:11:38.667237 4813 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:38.667307 4813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:38.667522 4813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:38.669649 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "97c3631784324193b450dc08f00aa23e"
format_stamp: "Formatted at 2026-09-23 18:11:38 on dist-test-slave-wsx5"
server_key: "675dbe6d417b6705aca022ffecac9661"
server_key_iv: "369b2143d75fc5994ab1298b3c12336c"
server_key_version: "encryptionkey@0"
I20260923 18:11:38.670207 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "97c3631784324193b450dc08f00aa23e"
format_stamp: "Formatted at 2026-09-23 18:11:38 on dist-test-slave-wsx5"
server_key: "675dbe6d417b6705aca022ffecac9661"
server_key_iv: "369b2143d75fc5994ab1298b3c12336c"
server_key_version: "encryptionkey@0"
I20260923 18:11:38.674168 4813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:11:38.676981 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:38.678395 4813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 18:11:38.678581 4813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "97c3631784324193b450dc08f00aa23e"
format_stamp: "Formatted at 2026-09-23 18:11:38 on dist-test-slave-wsx5"
server_key: "675dbe6d417b6705aca022ffecac9661"
server_key_iv: "369b2143d75fc5994ab1298b3c12336c"
server_key_version: "encryptionkey@0"
I20260923 18:11:38.678701 4813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:38.690686 4813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:38.691380 4813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:38.691560 4813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:38.692126 4813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:38.693095 4813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:38.693284 4813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:38.693429 4813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:38.693465 4813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:38.702029 4813 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:42113
I20260923 18:11:38.702052 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:42113 every 8 connection(s)
I20260923 18:11:38.702989 4813 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:38.709088 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 4813
I20260923 18:11:38.709267 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:11:38.709580 27951 external_mini_cluster.cc:1428] Setting key 4d7794476b514d2f868a08d5c686bc4b
I20260923 18:11:38.711481 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43015
--builtin_ntp_servers=127.27.75.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 18:11:38.712670 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:38.704967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:37915 (local address 127.27.75.254:43015)
0923 18:11:38.705093 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:38.705098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:38.706276 (+ 1178us) server_negotiation.cc:407] Connection header received
0923 18:11:38.707726 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:38.707730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:38.707795 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:38.707870 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:38.708930 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.709588 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.710772 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.710996 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.711087 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:38.712204 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:38.712208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:38.712210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:38.712223 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:38.712242 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:38.712250 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:38.712308 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:38.712452 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 18:11:38.712519 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 18:11:38.713693 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:38.705372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43015 (local address 127.27.75.193:37915)
0923 18:11:38.706114 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:38.706158 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:38.707469 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:38.708101 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:38.708110 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:38.708208 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:38.708774 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:38.708789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.709723 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.709726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:38.710646 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:38.710656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.711159 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.711163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:38.711325 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:38.711918 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:38.711920 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:38.711964 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:38.712048 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:38.712319 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:38.712329 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:38.712335 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:11:38.712447 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 18:11:38.712833 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":138,"threads_started":1}
I20260923 18:11:38.714929 4941 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43015
I20260923 18:11:38.715179 4941 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:38.715727 4941 heartbeater.cc:511] Master 127.27.75.254:43015 requested a full tablet report, sending...
I20260923 18:11:38.717284 4748 ts_manager.cc:194] Registered new tserver with Master: 97c3631784324193b450dc08f00aa23e (127.27.75.193:42113)
I20260923 18:11:38.718504 4748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.193:37915
W20260923 18:11:38.823014 4944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:38.823294 4944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:38.823352 4944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:38.827503 4944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:38.827567 4944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:38.827646 4944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:38.832590 4944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:38.833810 4944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:38.835219 4944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:38.845288 4949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:38.845280 4950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:38.845273 4952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:38.845587 4944 server_base.cc:1034] running on GCE node
I20260923 18:11:38.845995 4944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:38.846688 4944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:38.847862 4944 hybrid_clock.cc:648] HybridClock initialized: now 1790187098847839 us; error 33 us; skew 500 ppm
I20260923 18:11:38.849833 4944 webserver.cc:460] Webserver started at http://127.27.75.194:39615/ using document root <none> and password file <none>
I20260923 18:11:38.850374 4944 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:38.850435 4944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:38.850615 4944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:38.852641 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "6320b02e497749cda80615d0ced95323"
format_stamp: "Formatted at 2026-09-23 18:11:38 on dist-test-slave-wsx5"
server_key: "28cfc1a8bbd2e859e062ed09bca8e13c"
server_key_iv: "b9222aa05add08779cc6b2621bfdc434"
server_key_version: "encryptionkey@0"
I20260923 18:11:38.853192 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "6320b02e497749cda80615d0ced95323"
format_stamp: "Formatted at 2026-09-23 18:11:38 on dist-test-slave-wsx5"
server_key: "28cfc1a8bbd2e859e062ed09bca8e13c"
server_key_iv: "b9222aa05add08779cc6b2621bfdc434"
server_key_version: "encryptionkey@0"
I20260923 18:11:38.856920 4944 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:38.859603 4958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:38.860962 4944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:11:38.861079 4944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "6320b02e497749cda80615d0ced95323"
format_stamp: "Formatted at 2026-09-23 18:11:38 on dist-test-slave-wsx5"
server_key: "28cfc1a8bbd2e859e062ed09bca8e13c"
server_key_iv: "b9222aa05add08779cc6b2621bfdc434"
server_key_version: "encryptionkey@0"
I20260923 18:11:38.861234 4944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:38.874724 4944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:38.875427 4944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:38.875589 4944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:38.876138 4944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:38.877081 4944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:38.877154 4944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:38.877219 4944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:38.877242 4944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:38.886173 4944 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:42991
I20260923 18:11:38.886197 5071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:42991 every 8 connection(s)
I20260923 18:11:38.887101 4944 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:38.888160 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 4944
I20260923 18:11:38.888278 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:11:38.888636 27951 external_mini_cluster.cc:1428] Setting key 02e5eb8291f8c273ca48c7239682cb16
I20260923 18:11:38.890749 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43015
--builtin_ntp_servers=127.27.75.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 18:11:38.897877 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:38.889163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:40623 (local address 127.27.75.254:43015)
0923 18:11:38.889299 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:38.889303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:38.890574 (+ 1271us) server_negotiation.cc:407] Connection header received
0923 18:11:38.892161 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:38.892163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:38.892214 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:38.892283 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:38.893507 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.894063 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.895792 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.896009 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.896112 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:38.897409 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:38.897412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:38.897414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:38.897427 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:38.897446 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:38.897454 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:38.897511 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:38.897695 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 18:11:38.897757 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 18:11:38.899142 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:38.889657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43015 (local address 127.27.75.194:40623)
0923 18:11:38.890422 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:38.890469 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:38.891905 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:38.892480 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:38.892489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:38.892616 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:38.893329 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:38.893350 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.894197 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.894200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:38.895648 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:38.895661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:38.896198 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:38.896203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:38.896417 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:38.897053 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:38.897056 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:38.897144 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:38.897244 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:38.897533 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:38.897553 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:38.897559 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:11:38.897692 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 18:11:38.898030 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":154,"threads_started":1}
I20260923 18:11:38.900359 5072 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43015
I20260923 18:11:38.900591 5072 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:38.901217 5072 heartbeater.cc:511] Master 127.27.75.254:43015 requested a full tablet report, sending...
I20260923 18:11:38.902220 4748 ts_manager.cc:194] Registered new tserver with Master: 6320b02e497749cda80615d0ced95323 (127.27.75.194:42991)
I20260923 18:11:38.902822 4748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.194:40623
W20260923 18:11:39.003221 5075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:39.003484 5075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:39.003520 5075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:39.007732 5075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:39.007799 5075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:39.007858 5075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:39.012770 5075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:43015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:39.013927 5075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:39.015305 5075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:39.024904 5080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:39.024874 5083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:39.024890 5081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:39.025341 5075 server_base.cc:1034] running on GCE node
I20260923 18:11:39.025810 5075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:39.026530 5075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:39.027707 5075 hybrid_clock.cc:648] HybridClock initialized: now 1790187099027685 us; error 45 us; skew 500 ppm
I20260923 18:11:39.029641 5075 webserver.cc:460] Webserver started at http://127.27.75.195:36167/ using document root <none> and password file <none>
I20260923 18:11:39.030221 5075 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:39.030292 5075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:39.030506 5075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:39.032616 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "f2ce022e68084414b324c184c9271677"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "6f2e781ab7501cdcdffa30f981a593d9"
server_key_iv: "7550082e56fc6dc24587ef49084c2f6f"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.033273 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "f2ce022e68084414b324c184c9271677"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "6f2e781ab7501cdcdffa30f981a593d9"
server_key_iv: "7550082e56fc6dc24587ef49084c2f6f"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.037561 5075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:11:39.040360 5089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.041857 5075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:39.041999 5075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "f2ce022e68084414b324c184c9271677"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "6f2e781ab7501cdcdffa30f981a593d9"
server_key_iv: "7550082e56fc6dc24587ef49084c2f6f"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.042125 5075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:39.055678 5075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:39.056366 5075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:39.056543 5075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:39.057188 5075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:39.058169 5075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:39.058218 5075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.058274 5075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:39.058301 5075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.067009 5075 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:36739
I20260923 18:11:39.067076 5202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:36739 every 8 connection(s)
I20260923 18:11:39.067986 5075 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:39.077237 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 5075
I20260923 18:11:39.077351 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:11:39.077679 27951 external_mini_cluster.cc:1428] Setting key 450452309d7a36f6f5d01ad3ab8fb9f3
I20260923 18:11:39.077950 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:39.069983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:58813 (local address 127.27.75.254:43015)
0923 18:11:39.070135 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:39.070140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:39.071264 (+ 1124us) server_negotiation.cc:407] Connection header received
0923 18:11:39.072687 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:39.072691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:39.072747 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:39.072827 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:39.074007 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:39.074609 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:39.075825 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:39.076058 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:39.076147 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:39.077435 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:39.077441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:39.077444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:39.077464 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:39.077489 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:39.077503 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:39.077587 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:39.077728 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 18:11:39.077800 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:11:39.078828 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:39.070396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43015 (local address 127.27.75.195:58813)
0923 18:11:39.071108 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:39.071154 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:39.072440 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:39.073054 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:39.073062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:39.073198 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:39.073802 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:39.073847 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:39.074743 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:39.074746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:39.075695 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:39.075706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:39.076236 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:39.076239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:39.076386 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:39.076977 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:39.076979 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:39.077044 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:39.077202 (+ 158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:39.077589 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:39.077597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:39.077601 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 18:11:39.077706 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 18:11:39.077978 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":124,"threads_started":1}
I20260923 18:11:39.079893 5203 heartbeater.cc:347] Connected to a master server at 127.27.75.254:43015
I20260923 18:11:39.080134 5203 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:39.080643 5203 heartbeater.cc:511] Master 127.27.75.254:43015 requested a full tablet report, sending...
I20260923 18:11:39.081743 4748 ts_manager.cc:194] Registered new tserver with Master: f2ce022e68084414b324c184c9271677 (127.27.75.195:36739)
I20260923 18:11:39.082369 4748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.195:58813
I20260923 18:11:39.091933 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 18:11:39.094933 4792 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:39.093940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.27.75.254:43015)
0923 18:11:39.094080 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:39.094084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:39.094325 (+ 241us) server_negotiation.cc:407] Connection header received
0923 18:11:39.094406 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:39.094408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:39.094451 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:39.094523 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:39.094789 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39632: BlockingRecv error: recv got EOF from 127.0.0.1:39632 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:11:39.095073 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 4813
I20260923 18:11:39.099735 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 4944
I20260923 18:11:39.104699 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 5075
I20260923 18:11:39.109264 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 4722
2026-09-23T18:11:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (886 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T18:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:39Z Disabled control of system clock
I20260923 18:11:39.136476 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:33543
--webserver_interface=127.27.75.254
--webserver_port=0
--builtin_ntp_servers=127.27.75.212:36645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:33543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 18:11:39.245472 5214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:39.245769 5214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:39.245807 5214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:39.249886 5214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:39.249955 5214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:39.249969 5214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:39.249984 5214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:39.249998 5214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:39.254905 5214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:33543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:33543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:39.256009 5214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:39.257572 5214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:39.266033 5222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:39.266029 5220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:39.265995 5219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:39.266103 5214 server_base.cc:1034] running on GCE node
I20260923 18:11:39.266880 5214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:39.267990 5214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:39.269194 5214 hybrid_clock.cc:648] HybridClock initialized: now 1790187099269172 us; error 31 us; skew 500 ppm
I20260923 18:11:39.271016 5214 webserver.cc:460] Webserver started at http://127.27.75.254:46301/ using document root <none> and password file <none>
I20260923 18:11:39.271591 5214 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:39.271648 5214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:39.271812 5214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:39.273976 5214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "bbe4d1de44e7400eb014ee6dd1e5218b"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "2888c2670bb9eb1abcc4e982ca1d94ab"
server_key_iv: "582246548cc2cbef242fa4e908b3fdb4"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.274494 5214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "bbe4d1de44e7400eb014ee6dd1e5218b"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "2888c2670bb9eb1abcc4e982ca1d94ab"
server_key_iv: "582246548cc2cbef242fa4e908b3fdb4"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.278167 5214 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:11:39.280634 5228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.281993 5214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:11:39.282125 5214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "bbe4d1de44e7400eb014ee6dd1e5218b"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "2888c2670bb9eb1abcc4e982ca1d94ab"
server_key_iv: "582246548cc2cbef242fa4e908b3fdb4"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.282250 5214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:39.286995 5214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:39.287633 5214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:39.287791 5214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:39.294543 5214 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:33543
I20260923 18:11:39.294618 5280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:33543 every 8 connection(s)
I20260923 18:11:39.295547 5214 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:39.299079 5281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:39.301963 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 5214
I20260923 18:11:39.302073 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:39.302314 27951 external_mini_cluster.cc:1428] Setting key 02a2e84d2193c13096eec3a8e037be81
I20260923 18:11:39.305315 5281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b: Bootstrap starting.
I20260923 18:11:39.307785 5281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:39.308233 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:39.303749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.27.75.254:33543)
0923 18:11:39.304446 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:39.304462 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:39.304499 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:11:39.305218 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:39.305240 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:39.305645 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:39.305983 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:39.306867 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:39.306874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:39.306887 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:39.306919 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:39.306991 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:39.307011 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:39.307155 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:39.307229 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 18:11:39.307472 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":144,"threads_started":1}
I20260923 18:11:39.308619 5281 log.cc:824] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:39.310858 5281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b: No bootstrap required, opened a new log
I20260923 18:11:39.314442 5281 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe4d1de44e7400eb014ee6dd1e5218b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 33543 } }
I20260923 18:11:39.314683 5281 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:39.314718 5281 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe4d1de44e7400eb014ee6dd1e5218b, State: Initialized, Role: FOLLOWER
I20260923 18:11:39.315094 5281 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe4d1de44e7400eb014ee6dd1e5218b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 33543 } }
I20260923 18:11:39.315209 5281 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:39.315248 5281 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:39.315318 5281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:39.316224 5281 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe4d1de44e7400eb014ee6dd1e5218b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 33543 } }
I20260923 18:11:39.316540 5281 leader_election.cc:302] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbe4d1de44e7400eb014ee6dd1e5218b; no voters:
I20260923 18:11:39.316802 5281 leader_election.cc:288] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:39.316911 5286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:39.317183 5286 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [term 1 LEADER]: Becoming Leader. State: Replica: bbe4d1de44e7400eb014ee6dd1e5218b, State: Running, Role: LEADER
I20260923 18:11:39.317473 5286 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe4d1de44e7400eb014ee6dd1e5218b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 33543 } }
I20260923 18:11:39.317991 5281 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:39.319103 5288 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbe4d1de44e7400eb014ee6dd1e5218b. Latest consensus state: current_term: 1 leader_uuid: "bbe4d1de44e7400eb014ee6dd1e5218b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe4d1de44e7400eb014ee6dd1e5218b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 33543 } } }
I20260923 18:11:39.319226 5288 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:39.319561 5287 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbe4d1de44e7400eb014ee6dd1e5218b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe4d1de44e7400eb014ee6dd1e5218b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 33543 } } }
I20260923 18:11:39.319629 5295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:39.321218 5287 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:39.322785 5295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:39.329042 5295 catalog_manager.cc:1347] Generated new cluster ID: 91c68a803b864b06829a5ff85cc841c1
I20260923 18:11:39.329167 5295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:39.357249 5295 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:39.357676 5295 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:39.364985 5295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbe4d1de44e7400eb014ee6dd1e5218b: Generated new TSK 0
I20260923 18:11:39.365909 5295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:11:39.373418 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:33543
--builtin_ntp_servers=127.27.75.212:36645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 18:11:39.487214 5305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:39.487500 5305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:39.487560 5305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:39.491675 5305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:39.491747 5305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:39.491827 5305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:39.496683 5305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:39.497793 5305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:39.499162 5305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:11:39.508638 5305 server_base.cc:1034] running on GCE node
W20260923 18:11:39.508585 5310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:39.508608 5311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:39.508582 5313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:39.509407 5305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:39.510072 5305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:39.511247 5305 hybrid_clock.cc:648] HybridClock initialized: now 1790187099511225 us; error 38 us; skew 500 ppm
I20260923 18:11:39.513206 5305 webserver.cc:460] Webserver started at http://127.27.75.193:33459/ using document root <none> and password file <none>
I20260923 18:11:39.513834 5305 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:39.513908 5305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:39.514117 5305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:39.516212 5305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "a54eb9fa96ec4eaa94719dd29b024b73"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "90340ad14c3370c7ba3fcd6d52bfea6a"
server_key_iv: "15696d8b5c5447032ec747bd6320e61b"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.516778 5305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "a54eb9fa96ec4eaa94719dd29b024b73"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "90340ad14c3370c7ba3fcd6d52bfea6a"
server_key_iv: "15696d8b5c5447032ec747bd6320e61b"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.520696 5305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 18:11:39.523330 5319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.524637 5305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:11:39.524758 5305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "a54eb9fa96ec4eaa94719dd29b024b73"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "90340ad14c3370c7ba3fcd6d52bfea6a"
server_key_iv: "15696d8b5c5447032ec747bd6320e61b"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.524863 5305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:39.529501 5305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:39.530102 5305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:39.530260 5305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:39.530779 5305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:39.531714 5305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:39.531761 5305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.531813 5305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:39.531841 5305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.540707 5305 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:37885
I20260923 18:11:39.540737 5432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:37885 every 8 connection(s)
I20260923 18:11:39.541704 5305 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:39.548097 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:39.543711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:50923 (local address 127.27.75.254:33543)
0923 18:11:39.543873 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:39.543876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:39.545013 (+ 1137us) server_negotiation.cc:407] Connection header received
0923 18:11:39.546385 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:39.546389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:39.546440 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:39.546513 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:39.547644 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:39.547648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:39.547651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:39.547665 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:39.547686 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:39.547694 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:39.547780 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:39.547900 (+ 120us) server_negotiation.cc:299] Negotiation successful
0923 18:11:39.547951 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 18:11:39.548985 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:39.544109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:33543 (local address 127.27.75.193:50923)
0923 18:11:39.544834 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:39.544881 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:39.546153 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:39.546710 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:39.546718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:39.546820 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:39.547369 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:39.547370 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:39.547414 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:39.547485 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:39.547765 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:39.547774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:39.547779 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 18:11:39.547942 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 18:11:39.548234 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":122,"threads_started":1}
I20260923 18:11:39.549758 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 5305
I20260923 18:11:39.549860 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:11:39.550165 27951 external_mini_cluster.cc:1428] Setting key ba1e20fb66195aed9015e7477895c040
I20260923 18:11:39.550273 5433 heartbeater.cc:347] Connected to a master server at 127.27.75.254:33543
I20260923 18:11:39.550526 5433 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:39.550998 5433 heartbeater.cc:511] Master 127.27.75.254:33543 requested a full tablet report, sending...
I20260923 18:11:39.552143 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:33543
--builtin_ntp_servers=127.27.75.212:36645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 18:11:39.552788 5245 ts_manager.cc:194] Registered new tserver with Master: a54eb9fa96ec4eaa94719dd29b024b73 (127.27.75.193:37885)
W20260923 18:11:39.662966 5436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:39.663235 5436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:39.663275 5436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:39.667310 5436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:39.667378 5436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:39.667436 5436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:39.672274 5436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:39.673386 5436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:39.674811 5436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:39.684684 5444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:39.684785 5436 server_base.cc:1034] running on GCE node
W20260923 18:11:39.684787 5441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:39.684685 5442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:39.685494 5436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:39.686199 5436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:39.687415 5436 hybrid_clock.cc:648] HybridClock initialized: now 1790187099687364 us; error 62 us; skew 500 ppm
I20260923 18:11:39.689426 5436 webserver.cc:460] Webserver started at http://127.27.75.194:39719/ using document root <none> and password file <none>
I20260923 18:11:39.689991 5436 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:39.690054 5436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:39.690244 5436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:39.692297 5436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "b5be057bd2b7447f8ed664f7a00826e5"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "feb30169917b306a256e40862b69e9d0"
server_key_iv: "fbca9fe257c670af2a6a000ec7c62fd2"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.692873 5436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "b5be057bd2b7447f8ed664f7a00826e5"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "feb30169917b306a256e40862b69e9d0"
server_key_iv: "fbca9fe257c670af2a6a000ec7c62fd2"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.696805 5436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:11:39.699625 5450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.700928 5436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:11:39.701071 5436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "b5be057bd2b7447f8ed664f7a00826e5"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "feb30169917b306a256e40862b69e9d0"
server_key_iv: "fbca9fe257c670af2a6a000ec7c62fd2"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.701236 5436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:39.705919 5436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:39.706540 5436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:39.706699 5436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:39.707254 5436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:39.708153 5436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:39.708196 5436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.708240 5436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:39.708268 5436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.717180 5436 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:38403
I20260923 18:11:39.717227 5563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:38403 every 8 connection(s)
I20260923 18:11:39.718168 5436 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:39.718919 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 5436
I20260923 18:11:39.719050 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:11:39.719393 27951 external_mini_cluster.cc:1428] Setting key d4992b43bb511a400f446aac0143c3fa
I20260923 18:11:39.721462 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:33543
--builtin_ntp_servers=127.27.75.212:36645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 18:11:39.725163 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:39.720092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:54243 (local address 127.27.75.254:33543)
0923 18:11:39.720247 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:39.720252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:39.721629 (+ 1377us) server_negotiation.cc:407] Connection header received
0923 18:11:39.723335 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:39.723339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:39.723389 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:39.723466 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:39.724677 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:39.724681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:39.724683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:39.724696 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:39.724719 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:39.724728 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:39.724816 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:39.724957 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 18:11:39.724999 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 18:11:39.726732 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:39.720599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:33543 (local address 127.27.75.194:54243)
0923 18:11:39.721459 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:39.721523 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:39.723121 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:39.723649 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:39.723656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:39.723752 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:39.724326 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:39.724329 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:39.724385 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:39.724476 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:39.724817 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:39.724825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:39.724831 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:11:39.725006 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 18:11:39.725432 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260923 18:11:39.731410 5564 heartbeater.cc:347] Connected to a master server at 127.27.75.254:33543
I20260923 18:11:39.731727 5564 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:39.732340 5564 heartbeater.cc:511] Master 127.27.75.254:33543 requested a full tablet report, sending...
I20260923 18:11:39.733858 5245 ts_manager.cc:194] Registered new tserver with Master: b5be057bd2b7447f8ed664f7a00826e5 (127.27.75.194:38403)
W20260923 18:11:39.835081 5567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:39.835371 5567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:39.835428 5567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:39.839653 5567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:39.839726 5567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:39.839808 5567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:39.844791 5567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:39.845953 5567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:39.847340 5567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:39.857291 5575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:39.857295 5572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:39.857295 5573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:39.857627 5567 server_base.cc:1034] running on GCE node
I20260923 18:11:39.858143 5567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:39.858865 5567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:39.860044 5567 hybrid_clock.cc:648] HybridClock initialized: now 1790187099860024 us; error 32 us; skew 500 ppm
I20260923 18:11:39.862067 5567 webserver.cc:460] Webserver started at http://127.27.75.195:39517/ using document root <none> and password file <none>
I20260923 18:11:39.862679 5567 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:39.862751 5567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:39.862962 5567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:39.865011 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "740b93fb1a7445e9a77c933c408d135c"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "e2e7231921601dfd157712e910f8a7b2"
server_key_iv: "563f6aa88d0448db682fd5b3cf921262"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.865592 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "740b93fb1a7445e9a77c933c408d135c"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "e2e7231921601dfd157712e910f8a7b2"
server_key_iv: "563f6aa88d0448db682fd5b3cf921262"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.869321 5567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:11:39.872087 5581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.873461 5567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:39.873615 5567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "740b93fb1a7445e9a77c933c408d135c"
format_stamp: "Formatted at 2026-09-23 18:11:39 on dist-test-slave-wsx5"
server_key: "e2e7231921601dfd157712e910f8a7b2"
server_key_iv: "563f6aa88d0448db682fd5b3cf921262"
server_key_version: "encryptionkey@0"
I20260923 18:11:39.873733 5567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:39.878533 5567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:39.879164 5567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:39.879324 5567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:39.879856 5567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:39.880791 5567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:39.880836 5567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.880887 5567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:39.880913 5567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:39.889596 5567 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:37715
I20260923 18:11:39.889617 5694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:37715 every 8 connection(s)
I20260923 18:11:39.890517 5567 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:39.896862 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:39.892449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:35593 (local address 127.27.75.254:33543)
0923 18:11:39.892637 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:39.892642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:39.893915 (+ 1273us) server_negotiation.cc:407] Connection header received
0923 18:11:39.895178 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:39.895181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:39.895223 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:39.895310 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:39.896457 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:39.896460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:39.896462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:11:39.896475 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:39.896504 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:11:39.896513 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:39.896600 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:39.896708 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 18:11:39.896747 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 18:11:39.897768 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:39.892927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:33543 (local address 127.27.75.195:35593)
0923 18:11:39.893733 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:39.893779 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:39.894962 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:39.895456 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:39.895463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:39.895595 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:39.896157 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:11:39.896159 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:39.896203 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:11:39.896277 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:39.896595 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:39.896602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:39.896606 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 18:11:39.896739 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 18:11:39.896956 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":147,"threads_started":1}
I20260923 18:11:39.898370 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 5567
I20260923 18:11:39.898463 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:11:39.898823 27951 external_mini_cluster.cc:1428] Setting key c8cd09330b4a37d73f5d38c33ad28d98
I20260923 18:11:39.899012 5695 heartbeater.cc:347] Connected to a master server at 127.27.75.254:33543
I20260923 18:11:39.899242 5695 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:39.899775 5695 heartbeater.cc:511] Master 127.27.75.254:33543 requested a full tablet report, sending...
I20260923 18:11:39.900899 5244 ts_manager.cc:194] Registered new tserver with Master: 740b93fb1a7445e9a77c933c408d135c (127.27.75.195:37715)
I20260923 18:11:39.912904 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 18:11:39.915699 5284 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:11:39.914859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.27.75.254:33543)
0923 18:11:39.915021 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:39.915025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:39.915141 (+ 116us) server_negotiation.cc:407] Connection header received
0923 18:11:39.915245 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:39.915248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:39.915282 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:39.915345 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:39.915558 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60628: BlockingRecv error: recv got EOF from 127.0.0.1:60628 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 18:11:39.915774 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 5305
I20260923 18:11:39.920569 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 5436
I20260923 18:11:39.924996 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 5567
I20260923 18:11:39.929358 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 5214
2026-09-23T18:11:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (819 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:11:39 dist-test-slave-wsx5 krb5kdc[5705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:11:39 dist-test-slave-wsx5 krb5kdc[5705](info): set up 2 sockets
Sep 23 18:11:39 dist-test-slave-wsx5 krb5kdc[5705](info): commencing operation
krb5kdc: starting...
W20260923 18:11:40.884027 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:11:40 dist-test-slave-wsx5 krb5kdc[5705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:11:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:11:41.001978 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:35487
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:42147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:35487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:41.112114 5718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:41.112412 5718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:41.112454 5718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:41.116575 5718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:11:41.116653 5718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:41.116672 5718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:41.116688 5718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:11:41.116703 5718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:11:41.122244 5718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:42147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:35487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:35487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:41.123351 5718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:41.124362 5718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:41.133184 5723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:41.133230 5718 server_base.cc:1034] running on GCE node
W20260923 18:11:41.133177 5726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:41.133177 5724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:41.134061 5718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:41.135191 5718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:41.136368 5718 hybrid_clock.cc:648] HybridClock initialized: now 1790187101136348 us; error 35 us; skew 500 ppm
Sep 23 18:11:41 dist-test-slave-wsx5 krb5kdc[5705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187101, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:41.139220 5718 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:11:41.140167 5718 webserver.cc:460] Webserver started at http://127.27.75.254:43693/ using document root <none> and password file <none>
I20260923 18:11:41.140713 5718 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:41.140772 5718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:41.140964 5718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:41.143059 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "97ff60f87eb64f4fb5e164272cf74723"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "fd65ebf9969fdcc49b35847eff3de364"
server_key_iv: "67307e2e206790ad6abed1df00aae2ab"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.143611 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "97ff60f87eb64f4fb5e164272cf74723"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "fd65ebf9969fdcc49b35847eff3de364"
server_key_iv: "67307e2e206790ad6abed1df00aae2ab"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.147866 5718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:11:41.150542 5733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:41.151859 5718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:11:41.151989 5718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "97ff60f87eb64f4fb5e164272cf74723"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "fd65ebf9969fdcc49b35847eff3de364"
server_key_iv: "67307e2e206790ad6abed1df00aae2ab"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.152110 5718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:41.166257 5718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:41.169750 5718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:41.169958 5718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:41.176978 5718 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:35487
I20260923 18:11:41.177006 5785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:35487 every 8 connection(s)
I20260923 18:11:41.177992 5718 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:11:41.180771 5786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:41.186498 5786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723: Bootstrap starting.
I20260923 18:11:41.188267 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 5718
I20260923 18:11:41.188378 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:11:41.188632 27951 external_mini_cluster.cc:1428] Setting key d74fc1d3bcb5f6eeb11fae54d517c94e
I20260923 18:11:41.189150 5786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:41.190032 5786 log.cc:824] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:41.191947 5786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723: No bootstrap required, opened a new log
I20260923 18:11:41.195996 5786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ff60f87eb64f4fb5e164272cf74723" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35487 } }
I20260923 18:11:41.196301 5786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:41.196367 5786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ff60f87eb64f4fb5e164272cf74723, State: Initialized, Role: FOLLOWER
I20260923 18:11:41.196878 5786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ff60f87eb64f4fb5e164272cf74723" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35487 } }
I20260923 18:11:41.197045 5786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:11:41.197137 5786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:11:41.197237 5786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 18:11:41 dist-test-slave-wsx5 krb5kdc[5705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:41.198172 5786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ff60f87eb64f4fb5e164272cf74723" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35487 } }
I20260923 18:11:41.198544 5786 leader_election.cc:302] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97ff60f87eb64f4fb5e164272cf74723; no voters:
I20260923 18:11:41.198873 5786 leader_election.cc:288] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:11:41.198980 5791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:41.199216 5791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [term 1 LEADER]: Becoming Leader. State: Replica: 97ff60f87eb64f4fb5e164272cf74723, State: Running, Role: LEADER
I20260923 18:11:41.199550 5791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ff60f87eb64f4fb5e164272cf74723" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35487 } }
I20260923 18:11:41.200295 5786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:11:41.201081 5792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97ff60f87eb64f4fb5e164272cf74723. Latest consensus state: current_term: 1 leader_uuid: "97ff60f87eb64f4fb5e164272cf74723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ff60f87eb64f4fb5e164272cf74723" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35487 } } }
I20260923 18:11:41.201165 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97ff60f87eb64f4fb5e164272cf74723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ff60f87eb64f4fb5e164272cf74723" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35487 } } }
I20260923 18:11:41.201244 5792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:41.201264 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723 [sys.catalog]: This master's current role is: LEADER
I20260923 18:11:41.201637 5800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:11:41.204025 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.189884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.27.75.254:35487)
0923 18:11:41.190351 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:41.190367 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:41.190410 (+ 43us) server_negotiation.cc:407] Connection header received
0923 18:11:41.191076 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:41.191094 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:41.191423 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:41.191706 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:41.192583 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.193732 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.194903 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.195399 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.195524 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.198263 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:41.198270 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:41.198288 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:41.198321 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:41.200311 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:41.200846 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:41.200853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:41.200860 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:41.201031 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:41.201405 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:41.201408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:41.201410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:41.201941 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:41.202145 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:41.202908 (+ 763us) server_negotiation.cc:299] Negotiation successful
0923 18:11:41.203211 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
I20260923 18:11:41.204881 5800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:11:41.210315 5800 catalog_manager.cc:1347] Generated new cluster ID: def65fe7c722401081d01d5d7c4a8f70
I20260923 18:11:41.210435 5800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:11:41.239457 5800 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:11:41.240624 5800 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:11:41.245494 5800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723: Generated new TSK 0
I20260923 18:11:41.246256 5800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:11:41.284518 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35487
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:42147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:41.396541 5812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:41.396816 5812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:41.396867 5812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:41.401001 5812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:41.401069 5812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:41.401156 5812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:11:41.406576 5812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:42147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:41.407567 5812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:41.408530 5812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:11:41.418061 5812 server_base.cc:1034] running on GCE node
W20260923 18:11:41.418000 5818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:41.418008 5817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:41.417999 5820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:41.418588 5812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:41.419230 5812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:41.420419 5812 hybrid_clock.cc:648] HybridClock initialized: now 1790187101420399 us; error 33 us; skew 500 ppm
Sep 23 18:11:41 dist-test-slave-wsx5 krb5kdc[5705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187101, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:41.423661 5812 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:11:41.424638 5812 webserver.cc:460] Webserver started at http://127.27.75.193:43419/ using document root <none> and password file <none>
I20260923 18:11:41.425277 5812 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:41.425354 5812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:41.425561 5812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:41.427678 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "9302ef4cf4564342b1bb09aaf8f83278"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "61bf349b215ec7c4673c399ed3b7765c"
server_key_iv: "9aa9e84f60d04bbe734d688f508c6c93"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.428290 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "9302ef4cf4564342b1bb09aaf8f83278"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "61bf349b215ec7c4673c399ed3b7765c"
server_key_iv: "9aa9e84f60d04bbe734d688f508c6c93"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.432041 5812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 18:11:41.434705 5827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:41.435998 5812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:11:41.436133 5812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "9302ef4cf4564342b1bb09aaf8f83278"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "61bf349b215ec7c4673c399ed3b7765c"
server_key_iv: "9aa9e84f60d04bbe734d688f508c6c93"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.436247 5812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:41.452696 5812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:41.455720 5812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:41.455889 5812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:41.456475 5812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:41.457351 5812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:41.457398 5812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:41.457446 5812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:41.457466 5812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:41.465665 5812 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:41977
I20260923 18:11:41.465716 5940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:41977 every 8 connection(s)
I20260923 18:11:41.466606 5812 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:11:41.470947 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 5812
I20260923 18:11:41.471082 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:11:41.471385 27951 external_mini_cluster.cc:1428] Setting key 4b951eb10b74edee4d1613b4f99d5c76
Sep 23 18:11:41 dist-test-slave-wsx5 krb5kdc[5705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187101, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:11:41.481920 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.468438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:48967 (local address 127.27.75.254:35487)
0923 18:11:41.468634 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:41.468638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:41.469560 (+ 922us) server_negotiation.cc:407] Connection header received
0923 18:11:41.470984 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:41.470987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:41.471040 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:41.471128 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:41.472620 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.473247 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.474826 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.475068 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.475158 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.478175 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:41.478179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:41.478181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:41.478209 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:41.479616 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:41.480278 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:41.480281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:41.480282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:41.480372 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:41.480772 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:41.480775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:41.480776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:41.481000 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:41.481079 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:41.481642 (+ 563us) server_negotiation.cc:299] Negotiation successful
0923 18:11:41.481757 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 18:11:41.482810 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.468861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:35487 (local address 127.27.75.193:48967)
0923 18:11:41.469372 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:41.469419 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:41.470710 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:41.471376 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:41.471385 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:41.471854 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:41.472456 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.472476 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.473386 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.473389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.474704 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.474717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.475214 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.475218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.475384 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.475992 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:41.476031 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:41.477927 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:41.479757 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:41.479764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:41.479785 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:41.480157 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:41.480466 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:41.480469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:41.480471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:41.480675 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:41.481104 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:41.481111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:41.481412 (+ 301us) client_negotiation.cc:769] Sending connection context
0923 18:11:41.481593 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 18:11:41.481927 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:11:41.484151 5941 heartbeater.cc:347] Connected to a master server at 127.27.75.254:35487
I20260923 18:11:41.484411 5941 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:41.484975 5941 heartbeater.cc:511] Master 127.27.75.254:35487 requested a full tablet report, sending...
I20260923 18:11:41.486544 5750 ts_manager.cc:194] Registered new tserver with Master: 9302ef4cf4564342b1bb09aaf8f83278 (127.27.75.193:41977)
I20260923 18:11:41.487634 5750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:48967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:11:41.500727 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35487
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:42147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:41.610908 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:41.611207 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:41.611250 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:41.615613 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:41.615682 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:41.615744 5946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:11:41.621215 5946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:42147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:41.622301 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:41.623312 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:41.632856 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:41.632908 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:41.632856 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:41.633087 5946 server_base.cc:1034] running on GCE node
I20260923 18:11:41.633646 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:41.634289 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:41.635473 5946 hybrid_clock.cc:648] HybridClock initialized: now 1790187101635444 us; error 41 us; skew 500 ppm
Sep 23 18:11:41 dist-test-slave-wsx5 krb5kdc[5705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187101, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:41.638231 5946 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:11:41.639183 5946 webserver.cc:460] Webserver started at http://127.27.75.194:38803/ using document root <none> and password file <none>
I20260923 18:11:41.639802 5946 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:41.639874 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:41.640072 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:41.642223 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "b9e81f37e5ae42938a4b07c8fe097f23"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "067ba723b6749413fd7654fc87c7ac91"
server_key_iv: "a2bace48bef1094700a21da6df381efd"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.642784 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "b9e81f37e5ae42938a4b07c8fe097f23"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "067ba723b6749413fd7654fc87c7ac91"
server_key_iv: "a2bace48bef1094700a21da6df381efd"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.646586 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:11:41.649210 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:41.650483 5946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:11:41.650596 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "b9e81f37e5ae42938a4b07c8fe097f23"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "067ba723b6749413fd7654fc87c7ac91"
server_key_iv: "a2bace48bef1094700a21da6df381efd"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.650701 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:41.666802 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:41.670156 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:41.670336 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:41.670935 5946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:41.671751 5946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:41.671795 5946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:41.671836 5946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:41.671864 5946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:41.679994 5946 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:44827
I20260923 18:11:41.680061 6074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:44827 every 8 connection(s)
I20260923 18:11:41.680910 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:11:41.687248 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 5946
I20260923 18:11:41.687371 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:11:41.687629 27951 external_mini_cluster.cc:1428] Setting key 2c518d099c5ebe39d75c7ed6aded86bb
Sep 23 18:11:41 dist-test-slave-wsx5 krb5kdc[5705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187101, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:41.695170 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.682784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:50211 (local address 127.27.75.254:35487)
0923 18:11:41.682915 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:41.682918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:41.683758 (+ 840us) server_negotiation.cc:407] Connection header received
0923 18:11:41.685146 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:41.685150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:41.685206 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:41.685285 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:41.686548 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.687153 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.688473 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.688733 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.688835 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.691617 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:41.691627 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:41.691630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:41.691662 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:41.693059 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:41.693636 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:41.693639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:41.693640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:41.693722 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:41.694058 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:41.694061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:41.694063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:41.694264 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:41.694350 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:41.694870 (+ 520us) server_negotiation.cc:299] Negotiation successful
0923 18:11:41.695002 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:11:41.696051 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.683154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:35487 (local address 127.27.75.194:50211)
0923 18:11:41.683599 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:41.683642 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:41.684898 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:41.685491 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:41.685500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:41.685864 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:41.686383 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.686397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.687286 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.687288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.688362 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.688373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.688889 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.688893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.689152 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.689725 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:41.689743 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:41.691376 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:41.693200 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:41.693206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:41.693224 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:41.693532 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:41.693805 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:41.693808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:41.693810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:41.693960 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:41.694366 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:41.694374 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:41.694634 (+ 260us) client_negotiation.cc:769] Sending connection context
0923 18:11:41.694809 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 18:11:41.695177 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260923 18:11:41.697249 6075 heartbeater.cc:347] Connected to a master server at 127.27.75.254:35487
I20260923 18:11:41.697542 6075 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:41.698050 6075 heartbeater.cc:511] Master 127.27.75.254:35487 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
I20260923 18:11:41.699188 5750 ts_manager.cc:194] Registered new tserver with Master: b9e81f37e5ae42938a4b07c8fe097f23 (127.27.75.194:44827)
I20260923 18:11:41.699750 5750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:50211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:11:41.715723 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35487
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:42147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:11:41.832629 6080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:11:41.832896 6080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:11:41.832939 6080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:11:41.837062 6080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:11:41.837158 6080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:11:41.837226 6080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:11:41.842868 6080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:42147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:11:41.843993 6080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:11:41.845031 6080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:11:41.855140 6085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:41.855207 6088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:11:41.855185 6086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:11:41.855366 6080 server_base.cc:1034] running on GCE node
I20260923 18:11:41.855945 6080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:11:41.856726 6080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:11:41.857913 6080 hybrid_clock.cc:648] HybridClock initialized: now 1790187101857876 us; error 42 us; skew 500 ppm
Sep 23 18:11:41 dist-test-slave-wsx5 krb5kdc[5705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187101, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:11:41.860743 6080 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:11:41.861778 6080 webserver.cc:460] Webserver started at http://127.27.75.195:34623/ using document root <none> and password file <none>
I20260923 18:11:41.862331 6080 fs_manager.cc:359] Metadata directory not provided
I20260923 18:11:41.862398 6080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:11:41.862602 6080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:11:41.864750 6080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "13fa1d5d063e47feaa290ca8f3dbdba9"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "1259a1a427098572cdab3d260386345c"
server_key_iv: "bb74558cb5f6288b330c048e7c7f7ca2"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.865372 6080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "13fa1d5d063e47feaa290ca8f3dbdba9"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "1259a1a427098572cdab3d260386345c"
server_key_iv: "bb74558cb5f6288b330c048e7c7f7ca2"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.869410 6080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:11:41.872022 6095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:41.873363 6080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:11:41.873488 6080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "13fa1d5d063e47feaa290ca8f3dbdba9"
format_stamp: "Formatted at 2026-09-23 18:11:41 on dist-test-slave-wsx5"
server_key: "1259a1a427098572cdab3d260386345c"
server_key_iv: "bb74558cb5f6288b330c048e7c7f7ca2"
server_key_version: "encryptionkey@0"
I20260923 18:11:41.873603 6080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:11:41.889055 6080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:11:41.892585 6080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:11:41.892764 6080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:11:41.893417 6080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:11:41.894253 6080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:11:41.894299 6080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:41.894344 6080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:11:41.894372 6080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:11:41.902868 6080 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:43317
I20260923 18:11:41.902904 6208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:43317 every 8 connection(s)
I20260923 18:11:41.903779 6080 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:11:41.912580 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 6080
I20260923 18:11:41.912688 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:11:41.912972 27951 external_mini_cluster.cc:1428] Setting key 38738b8e0d23af58e781170c29ac1e76
Sep 23 18:11:41 dist-test-slave-wsx5 krb5kdc[5705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187101, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:11:41.917788 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.905720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:56617 (local address 127.27.75.254:35487)
0923 18:11:41.905889 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:41.905893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:41.906782 (+ 889us) server_negotiation.cc:407] Connection header received
0923 18:11:41.908206 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:41.908210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:41.908270 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:41.908346 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:41.909706 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.910243 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.911394 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.911618 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.911708 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.914458 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:41.914461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:41.914463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:41.914490 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:41.915767 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:41.916255 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:41.916258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:41.916259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:41.916343 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:41.916665 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:41.916669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:41.916671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:41.916849 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:41.916934 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:41.917497 (+ 563us) server_negotiation.cc:299] Negotiation successful
0923 18:11:41.917622 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 18:11:41.918614 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.906111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:35487 (local address 127.27.75.195:56617)
0923 18:11:41.906614 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:41.906660 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:41.907985 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:41.908535 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:41.908542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:41.908926 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:11:41.909555 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.909570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.910370 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.910374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.911262 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.911272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.911769 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.911773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.911949 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.912520 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:11:41.912536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:11:41.914240 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:11:41.915874 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:41.915881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:41.915898 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:41.916159 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:41.916425 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:11:41.916429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:11:41.916431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:11:41.916576 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:11:41.916958 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:11:41.916964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:11:41.917255 (+ 291us) client_negotiation.cc:769] Sending connection context
0923 18:11:41.917435 (+ 180us) client_negotiation.cc:240] Negotiation successful
0923 18:11:41.917775 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":133,"threads_started":1}
I20260923 18:11:41.919785 6209 heartbeater.cc:347] Connected to a master server at 127.27.75.254:35487
I20260923 18:11:41.920025 6209 heartbeater.cc:464] Registering TS with master...
I20260923 18:11:41.920569 6209 heartbeater.cc:511] Master 127.27.75.254:35487 requested a full tablet report, sending...
I20260923 18:11:41.921672 5750 ts_manager.cc:194] Registered new tserver with Master: 13fa1d5d063e47feaa290ca8f3dbdba9 (127.27.75.195:43317)
I20260923 18:11:41.922250 5750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:56617
I20260923 18:11:41.926846 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:11:41.936537 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.928756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.27.75.254:35487)
0923 18:11:41.928899 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:41.928904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:41.929082 (+ 178us) server_negotiation.cc:407] Connection header received
0923 18:11:41.929230 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:41.929233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:41.929284 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:41.929403 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:11:41.930020 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.930624 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.931485 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.931734 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.931841 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.933283 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:11:41.933286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:11:41.933288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:11:41.933312 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:11:41.934860 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:41.935315 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:41.935318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:41.935320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:41.935408 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:11:41.935656 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:11:41.935659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:11:41.935660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:11:41.935894 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:11:41.935996 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:41.936272 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 18:11:41.936395 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:11:41.939718 5750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:11:41.942874 5750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:11:41.957594 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.952876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.27.75.194:44827)
0923 18:11:41.953245 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:41.953256 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:41.953274 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:11:41.953332 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:41.953335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:41.953489 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:41.953616 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:41.953883 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.954910 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.956290 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.956895 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.956991 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.957175 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:41.957335 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 18:11:41.957431 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":148,"threads_started":1}
I20260923 18:11:41.957594 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.952469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:44827 (local address 127.0.0.1:34848)
0923 18:11:41.952762 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:41.952774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:41.952861 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:41.953617 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:41.953620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:41.953638 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:41.953759 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.953765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.955059 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.955063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.956192 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.956200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.957004 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.957008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.957114 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.957194 (+ 80us) client_negotiation.cc:769] Sending connection context
0923 18:11:41.957307 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 18:11:41.957447 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":50,"spinlock_wait_cycles":1920,"thread_start_us":79,"threads_started":1}
I20260923 18:11:41.957891 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.951506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.27.75.195:43317)
0923 18:11:41.951938 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:41.951950 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:41.951974 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:11:41.952131 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:41.952137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:41.952349 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:41.952504 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:41.952992 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.954149 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.955993 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.956954 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.957062 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.957221 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:41.957654 (+ 433us) server_negotiation.cc:299] Negotiation successful
0923 18:11:41.957754 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":103,"threads_started":1}
I20260923 18:11:41.957882 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.952137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:41977 (local address 127.0.0.1:35104)
0923 18:11:41.952757 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:41.952780 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:41.952884 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:41.953637 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:41.953640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:41.953658 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:41.953757 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.953763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.955401 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.955405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.956468 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.956480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.957538 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.957541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.957630 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.957651 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:11:41.957696 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 18:11:41.957760 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260923 18:11:41.958448 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.951422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:43317 (local address 127.0.0.1:42204)
0923 18:11:41.951779 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:41.951836 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:41.951967 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:41.952700 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:41.952705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:41.952728 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:41.952871 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.952878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.954309 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.954312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.955841 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:41.955852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.957284 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.957287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:41.957377 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.957400 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:11:41.957472 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:11:41.957548 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
I20260923 18:11:41.958799 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:41.952313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.27.75.193:41977)
0923 18:11:41.953088 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:41.953102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:41.953146 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:11:41.953219 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:41.953224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:41.953434 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:41.953888 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:41.954075 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.955079 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.956641 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:41.957411 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:41.958054 (+ 643us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:41.958191 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:41.958240 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 18:11:41.958330 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":67,"threads_started":1}
I20260923 18:11:41.960971 5875 tablet_service.cc:1467] Processing CreateTablet for tablet 6232dcf74f774920b988c58e80bea1f4 (DEFAULT_TABLE table=test-table [id=3648333d21b2472b858475ece01b2018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:41.961141 6143 tablet_service.cc:1467] Processing CreateTablet for tablet 6232dcf74f774920b988c58e80bea1f4 (DEFAULT_TABLE table=test-table [id=3648333d21b2472b858475ece01b2018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:41.961153 6009 tablet_service.cc:1467] Processing CreateTablet for tablet 6232dcf74f774920b988c58e80bea1f4 (DEFAULT_TABLE table=test-table [id=3648333d21b2472b858475ece01b2018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:11:41.962024 5875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6232dcf74f774920b988c58e80bea1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:41.962149 6009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6232dcf74f774920b988c58e80bea1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:41.962154 6143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6232dcf74f774920b988c58e80bea1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:11:41.968403 6226 tablet_bootstrap.cc:492] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278: Bootstrap starting.
I20260923 18:11:41.970023 6227 tablet_bootstrap.cc:492] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: Bootstrap starting.
I20260923 18:11:41.971179 6226 tablet_bootstrap.cc:654] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:41.972458 6226 log.cc:824] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:41.972687 6227 tablet_bootstrap.cc:654] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:41.973076 6228 tablet_bootstrap.cc:492] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: Bootstrap starting.
I20260923 18:11:41.973551 6227 log.cc:824] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:41.975229 6227 tablet_bootstrap.cc:492] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: No bootstrap required, opened a new log
I20260923 18:11:41.975245 6226 tablet_bootstrap.cc:492] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278: No bootstrap required, opened a new log
I20260923 18:11:41.975418 6227 ts_tablet_manager.cc:1397] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:11:41.975420 6226 ts_tablet_manager.cc:1397] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 18:11:41.975833 6228 tablet_bootstrap.cc:654] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: Neither blocks nor log segments found. Creating new log.
I20260923 18:11:41.976639 6228 log.cc:824] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: Log is configured to *not* fsync() on all Append() calls
I20260923 18:11:41.978658 6227 raft_consensus.cc:348] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43317 } } peers { permanent_uuid: "9302ef4cf4564342b1bb09aaf8f83278" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41977 } } peers { permanent_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44827 } }
I20260923 18:11:41.978849 6228 tablet_bootstrap.cc:492] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: No bootstrap required, opened a new log
I20260923 18:11:41.978880 6227 raft_consensus.cc:374] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:41.978932 6227 raft_consensus.cc:729] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e81f37e5ae42938a4b07c8fe097f23, State: Initialized, Role: FOLLOWER
I20260923 18:11:41.979033 6228 ts_tablet_manager.cc:1397] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:11:41.979137 6226 raft_consensus.cc:348] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43317 } } peers { permanent_uuid: "9302ef4cf4564342b1bb09aaf8f83278" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41977 } } peers { permanent_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44827 } }
I20260923 18:11:41.979384 6226 raft_consensus.cc:374] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:41.979444 6226 raft_consensus.cc:729] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9302ef4cf4564342b1bb09aaf8f83278, State: Initialized, Role: FOLLOWER
I20260923 18:11:41.979383 6227 consensus_queue.cc:260] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43317 } } peers { permanent_uuid: "9302ef4cf4564342b1bb09aaf8f83278" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41977 } } peers { permanent_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44827 } }
I20260923 18:11:41.979856 6226 consensus_queue.cc:260] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43317 } } peers { permanent_uuid: "9302ef4cf4564342b1bb09aaf8f83278" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41977 } } peers { permanent_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44827 } }
I20260923 18:11:41.980038 6075 heartbeater.cc:503] Master 127.27.75.254:35487 was elected leader, sending a full tablet report...
I20260923 18:11:41.980808 6226 ts_tablet_manager.cc:1428] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 18:11:41.981176 5941 heartbeater.cc:503] Master 127.27.75.254:35487 was elected leader, sending a full tablet report...
I20260923 18:11:41.981576 6227 ts_tablet_manager.cc:1428] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:11:41.982887 6228 raft_consensus.cc:348] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43317 } } peers { permanent_uuid: "9302ef4cf4564342b1bb09aaf8f83278" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41977 } } peers { permanent_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44827 } }
I20260923 18:11:41.983248 6228 raft_consensus.cc:374] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:11:41.983310 6228 raft_consensus.cc:729] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13fa1d5d063e47feaa290ca8f3dbdba9, State: Initialized, Role: FOLLOWER
I20260923 18:11:41.983762 6228 consensus_queue.cc:260] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43317 } } peers { permanent_uuid: "9302ef4cf4564342b1bb09aaf8f83278" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41977 } } peers { permanent_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44827 } }
I20260923 18:11:41.984691 6209 heartbeater.cc:503] Master 127.27.75.254:35487 was elected leader, sending a full tablet report...
I20260923 18:11:41.984762 6228 ts_tablet_manager.cc:1428] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 18:11:42.151619 6233 raft_consensus.cc:482] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:11:42.151823 6233 raft_consensus.cc:504] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43317 } } peers { permanent_uuid: "9302ef4cf4564342b1bb09aaf8f83278" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41977 } } peers { permanent_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44827 } }
I20260923 18:11:42.152944 6233 leader_election.cc:288] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13fa1d5d063e47feaa290ca8f3dbdba9 (127.27.75.195:43317), b9e81f37e5ae42938a4b07c8fe097f23 (127.27.75.194:44827)
W20260923 18:11:42.154675 6210 tablet.cc:2369] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:11:42.156809 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:42.153373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:57661 (local address 127.27.75.195:43317)
0923 18:11:42.153541 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:42.153546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:42.153643 (+ 97us) server_negotiation.cc:407] Connection header received
0923 18:11:42.153792 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:42.153795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:42.153851 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:42.153924 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:42.154175 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:42.154720 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:42.155835 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:42.156300 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:42.156412 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:42.156468 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:42.156624 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 18:11:42.156697 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 18:11:42.156824 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:42.153237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:43317 (local address 127.27.75.193:57661)
0923 18:11:42.153540 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:42.153554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:42.153691 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:42.153944 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:42.153947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:42.153968 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:42.154080 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:42.154086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:42.154887 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:42.154891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:42.155718 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:42.155728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:42.156411 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:42.156415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:42.156517 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:42.156536 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 18:11:42.156599 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 18:11:42.156670 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":63,"threads_started":1}
I20260923 18:11:42.157269 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:42.153625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:44827 (local address 127.27.75.193:41877)
0923 18:11:42.153938 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:42.153948 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 18:11:42.154031 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:11:42.154232 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:11:42.154234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:11:42.154248 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:11:42.154364 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:42.154369 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:42.155334 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:42.155337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:42.156191 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:11:42.156200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:11:42.156813 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:11:42.156817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:11:42.156913 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:42.156937 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:11:42.157039 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 18:11:42.157108 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":98,"threads_started":1}
I20260923 18:11:42.157284 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:42.153752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:41877 (local address 127.27.75.194:44827)
0923 18:11:42.153866 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:42.153870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:42.154013 (+ 143us) server_negotiation.cc:407] Connection header received
0923 18:11:42.154112 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:42.154115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:42.154156 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:42.154217 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:11:42.154530 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:42.155230 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:42.156302 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:42.156680 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:42.156808 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:42.156858 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:42.157074 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 18:11:42.157173 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 18:11:42.157533 6163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6232dcf74f774920b988c58e80bea1f4" candidate_uuid: "9302ef4cf4564342b1bb09aaf8f83278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" is_pre_election: true
I20260923 18:11:42.157838 6163 raft_consensus.cc:2454] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9302ef4cf4564342b1bb09aaf8f83278 in term 0.
I20260923 18:11:42.158278 6029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6232dcf74f774920b988c58e80bea1f4" candidate_uuid: "9302ef4cf4564342b1bb09aaf8f83278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" is_pre_election: true
I20260923 18:11:42.158350 5831 leader_election.cc:302] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13fa1d5d063e47feaa290ca8f3dbdba9, 9302ef4cf4564342b1bb09aaf8f83278; no voters:
I20260923 18:11:42.158547 6029 raft_consensus.cc:2454] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9302ef4cf4564342b1bb09aaf8f83278 in term 0.
I20260923 18:11:42.158653 6233 raft_consensus.cc:2781] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:11:42.158748 6233 raft_consensus.cc:482] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:11:42.158799 6233 raft_consensus.cc:3037] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:42.159808 6233 raft_consensus.cc:504] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43317 } } peers { permanent_uuid: "9302ef4cf4564342b1bb09aaf8f83278" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41977 } } peers { permanent_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44827 } }
I20260923 18:11:42.160130 6233 leader_election.cc:288] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [CANDIDATE]: Term 1 election: Requested vote from peers 13fa1d5d063e47feaa290ca8f3dbdba9 (127.27.75.195:43317), b9e81f37e5ae42938a4b07c8fe097f23 (127.27.75.194:44827)
I20260923 18:11:42.160526 6029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6232dcf74f774920b988c58e80bea1f4" candidate_uuid: "9302ef4cf4564342b1bb09aaf8f83278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e81f37e5ae42938a4b07c8fe097f23"
I20260923 18:11:42.160544 6163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6232dcf74f774920b988c58e80bea1f4" candidate_uuid: "9302ef4cf4564342b1bb09aaf8f83278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9"
I20260923 18:11:42.160627 6029 raft_consensus.cc:3037] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:42.160655 6163 raft_consensus.cc:3037] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:11:42.161715 6163 raft_consensus.cc:2454] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9302ef4cf4564342b1bb09aaf8f83278 in term 1.
I20260923 18:11:42.161716 6029 raft_consensus.cc:2454] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9302ef4cf4564342b1bb09aaf8f83278 in term 1.
I20260923 18:11:42.161978 5831 leader_election.cc:302] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13fa1d5d063e47feaa290ca8f3dbdba9, 9302ef4cf4564342b1bb09aaf8f83278; no voters:
I20260923 18:11:42.162137 6233 raft_consensus.cc:2781] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:11:42.162374 6233 raft_consensus.cc:686] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [term 1 LEADER]: Becoming Leader. State: Replica: 9302ef4cf4564342b1bb09aaf8f83278, State: Running, Role: LEADER
I20260923 18:11:42.162711 6233 consensus_queue.cc:237] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43317 } } peers { permanent_uuid: "9302ef4cf4564342b1bb09aaf8f83278" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41977 } } peers { permanent_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44827 } }
I20260923 18:11:42.165802 5750 catalog_manager.cc:5495] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9302ef4cf4564342b1bb09aaf8f83278 (127.27.75.193). New cstate: current_term: 1 leader_uuid: "9302ef4cf4564342b1bb09aaf8f83278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9302ef4cf4564342b1bb09aaf8f83278" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44827 } health_report { overall_health: UNKNOWN } } }
W20260923 18:11:42.182056 6076 tablet.cc:2369] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:11:42.193485 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:42.190075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.27.75.193:41977)
0923 18:11:42.190187 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 18:11:42.190191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:11:42.190203 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:11:42.190273 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:11:42.190275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:11:42.190328 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:11:42.190412 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:11:42.190632 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:42.191123 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:42.192014 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:11:42.192235 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:11:42.192317 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:11:42.192567 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:11:42.193000 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:11:42.193093 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:11:42.193250 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 18:11:42.193306 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 18:11:42.199541 6163 raft_consensus.cc:1261] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9 [term 1 FOLLOWER]: Refusing update from remote peer 9302ef4cf4564342b1bb09aaf8f83278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:42.199518 6029 raft_consensus.cc:1261] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23 [term 1 FOLLOWER]: Refusing update from remote peer 9302ef4cf4564342b1bb09aaf8f83278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:11:42.200356 6233 consensus_queue.cc:1035] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13fa1d5d063e47feaa290ca8f3dbdba9" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:42.200656 6237 consensus_queue.cc:1035] T 6232dcf74f774920b988c58e80bea1f4 P 9302ef4cf4564342b1bb09aaf8f83278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9e81f37e5ae42938a4b07c8fe097f23" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:11:42.211035 6242 mvcc.cc:205] Tried to move back new op lower bound from 7332606370601558016 to 7332606370461192192. Current Snapshot: MvccSnapshot[applied={T|T < 7332606370601558016}]
I20260923 18:11:42.217978 5750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:42.218179 5750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:11:42.220652 5750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97ff60f87eb64f4fb5e164272cf74723: Sending DeleteTablet for 3 replicas of tablet 6232dcf74f774920b988c58e80bea1f4
I20260923 18:11:42.221565 6143 tablet_service.cc:1514] Processing DeleteTablet for tablet 6232dcf74f774920b988c58e80bea1f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:42 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:42204
I20260923 18:11:42.221690 6009 tablet_service.cc:1514] Processing DeleteTablet for tablet 6232dcf74f774920b988c58e80bea1f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:11:42 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:34848
I20260923 18:11:42.222111 6249 tablet_replica.cc:330] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: stopping tablet replica
I20260923 18:11:42.222200 6250 tablet_replica.cc:330] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: stopping tablet replica
I20260923 18:11:42.222345 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 5812
I20260923 18:11:42.222445 6250 raft_consensus.cc:2229] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:42.222446 6249 raft_consensus.cc:2229] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:11:42.222770 6249 raft_consensus.cc:2258] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:42.222770 6250 raft_consensus.cc:2258] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:11:42.224365 6250 ts_tablet_manager.cc:1905] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:42.224365 6249 ts_tablet_manager.cc:1905] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:11:42.227123 6250 ts_tablet_manager.cc:1918] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:42.227123 6249 ts_tablet_manager.cc:1918] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:11:42.227231 6249 log.cc:1196] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/6232dcf74f774920b988c58e80bea1f4
I20260923 18:11:42.227231 6250 log.cc:1196] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/6232dcf74f774920b988c58e80bea1f4
I20260923 18:11:42.227535 6249 ts_tablet_manager.cc:1939] T 6232dcf74f774920b988c58e80bea1f4 P 13fa1d5d063e47feaa290ca8f3dbdba9: Deleting consensus metadata
I20260923 18:11:42.227543 6250 ts_tablet_manager.cc:1939] T 6232dcf74f774920b988c58e80bea1f4 P b9e81f37e5ae42938a4b07c8fe097f23: Deleting consensus metadata
I20260923 18:11:42.228917 5737 catalog_manager.cc:4841] TS 13fa1d5d063e47feaa290ca8f3dbdba9 (127.27.75.195:43317): tablet 6232dcf74f774920b988c58e80bea1f4 (table test-table [id=3648333d21b2472b858475ece01b2018]) successfully deleted
I20260923 18:11:42.229115 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 5946
I20260923 18:11:42.229169 5737 catalog_manager.cc:4841] TS b9e81f37e5ae42938a4b07c8fe097f23 (127.27.75.194:44827): tablet 6232dcf74f774920b988c58e80bea1f4 (table test-table [id=3648333d21b2472b858475ece01b2018]) successfully deleted
W20260923 18:11:42.229322 5737 connection.cc:535] client connection to 127.27.75.193:41977 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 18:11:42.229432 5737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 18:11:42.232139 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:11:42.231832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:41977 (local address 127.0.0.1:35132)
0923 18:11:42.231967 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 18:11:42.232043 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 18:11:42.232347 5737 catalog_manager.cc:4584] TS 9302ef4cf4564342b1bb09aaf8f83278 (127.27.75.193:41977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6232dcf74f774920b988c58e80bea1f4: Network error: Client connection negotiation failed: client connection to 127.27.75.193:41977: connect: Connection refused (error 111)
I20260923 18:11:42.234313 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 6080
I20260923 18:11:42.239204 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 5718
2026-09-23T18:11:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2311 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:11:42 dist-test-slave-wsx5 krb5kdc[6254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:11:42 dist-test-slave-wsx5 krb5kdc[6254](info): set up 2 sockets
Sep 23 18:11:42 dist-test-slave-wsx5 krb5kdc[6254](info): commencing operation
krb5kdc: starting...
W20260923 18:11:43.204157 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:11:43 dist-test-slave-wsx5 krb5kdc[6254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:11:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/rangeradmin_127.27.75.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/rangeradmin_127.27.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/rangerlookup_127.27.75.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/rangerlookup_127.27.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/HTTP_127.27.75.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/HTTP_127.27.75.212@KRBTEST.COM.keytab.
I20260923 18:11:43.376489 27951 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknO4ss8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/postgres -l logfile start
2026-09-23 18:11:45.976 UTC [6284] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 18:11:45.976 UTC [6284] LOG: listening on IPv4 address "127.27.75.212", port 53327
2026-09-23 18:11:45.981 UTC [6284] LOG: listening on Unix socket "/tmp/.s.PGSQL.53327"
2026-09-23 18:11:46.011 UTC [6287] LOG: database system was shut down at 2026-09-23 18:11:44 UTC
2026-09-23 18:11:46.015 UTC [6284] LOG: database system is ready to accept connections
I20260923 18:11:46.888064 27951 mini_postgres.cc:96] Postgres bound to 53327
2026-09-23 18:11:46.894 UTC [6295] FATAL: database "slave" does not exist
127.27.75.212:53327 - accepting connections
I20260923 18:11:46.894910 27951 mini_ranger.cc:161] Starting Ranger...
I20260923 18:11:46.910075 27951 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 18:11:47.483225 27951 mini_ranger.cc:87] Created ranger Postgres database
I20260923 18:11:47.483322 27951 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-admin
2026-09-23 18:11:47,774 [I] DB FLAVOR :POSTGRES
2026-09-23 18:11:47,775 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 18:11:47,775 [I] Checking connection..
2026-09-23 18:11:47,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 18:11:48,029 [I] Checking connection passed.
2026-09-23 18:11:48,029 [I] --------- Verifying version history table ---------
2026-09-23 18:11:48,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:11:48,296 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 18:11:48,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:11:48,539 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 18:11:48,540 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 18:11:48,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 18:11:48.798 UTC [6359] WARNING: there is no transaction in progress
2026-09-23 18:11:48,804 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 18:11:48,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:11:49,048 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 18:11:49,048 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 18:11:49,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 18:11:49,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 18:11:49,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wsx5.c.gcp-upstream.internal','N') ;"
2026-09-23 18:11:49,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 18:11:50,050 [I] Table x_portal_user does not exist in database ranger
2026-09-23 18:11:50,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 18:11:50,299 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 18:11:50,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 18:11:50,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 18:11:50,798 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 18:11:50,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 18:11:51.833 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:51.851 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:51.869 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:51.886 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:51.903 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:51.924 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:51.988 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:51.996 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:52.011 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:52.023 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:52.034 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:52.050 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:52.060 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:52.068 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:52.076 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:52.083 UTC [6467] WARNING: there is no transaction in progress
2026-09-23 18:11:52,813 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 18:11:52,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 18:11:53,066 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 18:11:53,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 18:11:53,311 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 18:11:53,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 18:11:53,562 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 18:11:53,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 18:11:53,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 18:11:54,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wsx5.c.gcp-upstream.internal';"
2026-09-23 18:11:54,332 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 18:11:54,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 18:11:54,599 [I] DB_PATCHES have already been applied
I20260923 18:11:54.604645 27951 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknO4ss8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 18:11:54.604712 27951 mini_ranger.cc:193] Using host: 127.27.75.212
I20260923 18:11:54.607971 27951 mini_ranger.cc:227] Ranger admin URL: http://127.27.75.212:56889
Sep 23, 2026 6:11:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 6:11:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 6:11:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 6:11:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 6:11:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 6:11:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 6:11:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 18:11:55.596930 27951 mini_ranger.cc:160] Time spent starting Ranger: real 8.702s user 0.000s sys 0.010s
Sep 23, 2026 6:12:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 6:12:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 6:12:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 6:12:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 6:12:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 18:12:18.610999 27951 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/rangerkms_127.27.75.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/rangerkms_127.27.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.75.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.75.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.75.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/HTTP_127.27.75.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.75.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/HTTP_127.27.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 18:12:18 dist-test-slave-wsx5 krb5kdc[6254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187138, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 18:12:18.695614 27951 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 18:12:18.705312 27951 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 18:12:19.641531 27951 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 18:12:19.641623 27951 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-kms
I20260923 18:12:19.641721 27951 mini_ranger_kms.cc:227] Using postgres at 127.27.75.212:53327
2026-09-23 18:12:19,823 [I] DB FLAVOR :POSTGRES
2026-09-23 18:12:19,824 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 18:12:19,824 [I] Checking connection
2026-09-23 18:12:19,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 18:12:20,063 [I] connection success
2026-09-23 18:12:20,064 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 18:12:20,064 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 18:12:20,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 18:12:20,312 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 18:12:20,312 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 18:12:20,312 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 18:12:20,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:53327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 18:12:20,577 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 18:12:21.085678 27951 mini_ranger_kms.cc:310] Created kms service
I20260923 18:12:21.275310 27951 mini_ranger_kms.cc:326] Created kudu user
I20260923 18:12:21.322305 27951 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 18:12:21 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187138, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:12:21.791675 27951 mini_ranger_kms.cc:384] Added ranger policy
I20260923 18:12:21.792007 27951 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 18:12:21.792040 27951 mini_ranger_kms.cc:240] Using host: 127.27.75.212
I20260923 18:12:21.796058 27951 mini_ranger_kms.cc:276] Ranger KMS PID: 6648
I20260923 18:12:21.796195 27951 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.75.212:43649
Sep 23, 2026 6:12:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 6:12:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 6:12:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 6:12:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 6:12:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 6:12:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 18:12:23.859958 27951 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.164s user 0.001s sys 0.166s
I20260923 18:12:23.860085 27951 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 18:12:23.860185 27951 mini_ranger_kms.cc:401] 127.27.75.212:43649/kms/v1/keys
Sep 23 18:12:23 dist-test-slave-wsx5 krb5kdc[6254](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790187143, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.75.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 18:12:24 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187143, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.75.212@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
Sep 23 18:12:24 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187143, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.75.212@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
Sep 23 18:12:27 dist-test-slave-wsx5 krb5kdc[6254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:12:27.166841 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34941
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:34941
--ranger_config_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:43649/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:12:27.280412 6698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:12:27.280709 6698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:12:27.280767 6698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:12:27.284991 6698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:12:27.285079 6698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:12:27.285104 6698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:12:27.285149 6698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:12:27.285179 6698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:12:27.290822 6698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.75.212:43649/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:34941
--ranger_config_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:12:27.291956 6698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:12:27.293043 6698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:12:27.302029 6706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:12:27.302148 6698 server_base.cc:1034] running on GCE node
W20260923 18:12:27.302054 6704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:12:27.302055 6703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:12:27.303329 6698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:12:27.304610 6698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:12:27.305838 6698 hybrid_clock.cc:648] HybridClock initialized: now 1790187147305783 us; error 74 us; skew 500 ppm
Sep 23 18:12:27 dist-test-slave-wsx5 krb5kdc[6254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187147, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:12:27.308930 6698 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:12:27.309976 6698 webserver.cc:460] Webserver started at http://127.27.75.254:38791/ using document root <none> and password file <none>
I20260923 18:12:27.310513 6698 fs_manager.cc:359] Metadata directory not provided
I20260923 18:12:27.310570 6698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:12:27.310779 6698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:12:27 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187147, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:12:27.367913 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "fbfad4a0a8d142648f67236125203127"
format_stamp: "Formatted at 2026-09-23 18:12:27 on dist-test-slave-wsx5"
server_key: "ef1b2d132dd6af28066eb4fac89e43f8"
server_key_iv: "7adec9fd20c64fe0b5b079acb5dcce88"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:27.368533 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "fbfad4a0a8d142648f67236125203127"
format_stamp: "Formatted at 2026-09-23 18:12:27 on dist-test-slave-wsx5"
server_key: "ef1b2d132dd6af28066eb4fac89e43f8"
server_key_iv: "7adec9fd20c64fe0b5b079acb5dcce88"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:27.372495 6698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:12:27.391266 6713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:12:27.392637 6698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:12:27.392777 6698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "fbfad4a0a8d142648f67236125203127"
format_stamp: "Formatted at 2026-09-23 18:12:27 on dist-test-slave-wsx5"
server_key: "ef1b2d132dd6af28066eb4fac89e43f8"
server_key_iv: "7adec9fd20c64fe0b5b079acb5dcce88"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:27.392900 6698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:12:27.407753 6698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:12:27.410480 6698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:12:27.410652 6698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:12:27.418025 6698 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:34941
I20260923 18:12:27.418036 6765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:34941 every 8 connection(s)
I20260923 18:12:27.418979 6698 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:12:27.421808 6766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:12:27.425437 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 6698
I20260923 18:12:27.425558 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:12:27.425843 27951 external_mini_cluster.cc:1428] Setting key c531073907fc85022c449ed0e2b469d2
I20260923 18:12:27.427924 6766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127: Bootstrap starting.
I20260923 18:12:27.430265 6766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127: Neither blocks nor log segments found. Creating new log.
I20260923 18:12:27.431013 6766 log.cc:824] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127: Log is configured to *not* fsync() on all Append() calls
I20260923 18:12:27.432876 6766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127: No bootstrap required, opened a new log
Sep 23 18:12:27 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:12:27.436704 6766 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfad4a0a8d142648f67236125203127" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34941 } }
I20260923 18:12:27.436956 6766 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:12:27.437014 6766 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfad4a0a8d142648f67236125203127, State: Initialized, Role: FOLLOWER
I20260923 18:12:27.437503 6766 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfad4a0a8d142648f67236125203127" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34941 } }
I20260923 18:12:27.437637 6766 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:12:27.437703 6766 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:12:27.437799 6766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:12:27.438670 6766 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfad4a0a8d142648f67236125203127" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34941 } }
I20260923 18:12:27.439016 6766 leader_election.cc:302] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbfad4a0a8d142648f67236125203127; no voters:
I20260923 18:12:27.439347 6766 leader_election.cc:288] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:12:27.439429 6771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:12:27.439628 6771 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [term 1 LEADER]: Becoming Leader. State: Replica: fbfad4a0a8d142648f67236125203127, State: Running, Role: LEADER
I20260923 18:12:27.440001 6771 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfad4a0a8d142648f67236125203127" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34941 } }
I20260923 18:12:27.440387 6769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:27.427327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.27.75.254:34941)
0923 18:12:27.427822 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0923 18:12:27.427839 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 18:12:27.427878 (+ 39us) server_negotiation.cc:407] Connection header received
0923 18:12:27.428610 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:12:27.428630 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:12:27.428975 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:12:27.429346 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:12:27.430189 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:27.431283 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:27.432173 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:27.432651 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:27.432803 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:27.435480 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:12:27.435485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:12:27.435500 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:12:27.435530 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:12:27.437216 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:12:27.437791 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:12:27.437797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:12:27.437803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:12:27.437931 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:12:27.438191 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:12:27.438194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:12:27.438196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:12:27.438610 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:12:27.438790 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 18:12:27.439085 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 18:12:27.439410 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":145,"threads_started":1}
I20260923 18:12:27.440563 6766 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:12:27.442338 6773 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbfad4a0a8d142648f67236125203127. Latest consensus state: current_term: 1 leader_uuid: "fbfad4a0a8d142648f67236125203127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfad4a0a8d142648f67236125203127" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34941 } } }
I20260923 18:12:27.442473 6773 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [sys.catalog]: This master's current role is: LEADER
I20260923 18:12:27.442885 6772 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbfad4a0a8d142648f67236125203127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfad4a0a8d142648f67236125203127" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34941 } } }
I20260923 18:12:27.443027 6772 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127 [sys.catalog]: This master's current role is: LEADER
I20260923 18:12:27.443363 6781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:12:27.443449 6766 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 18:12:27.448889 6781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:12:27.455732 6781 catalog_manager.cc:1347] Generated new cluster ID: 072d24ae63514907ba8c44e40bd27a14
I20260923 18:12:27.455813 6781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:12:27.469189 6781 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:12:27.471166 6781 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:12:27.478703 6781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127: Generated new TSK 0
I20260923 18:12:27.479562 6781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:12:28.203833 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:43649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34941
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:12:28.361640 6811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:12:28.362080 6811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:12:28.362147 6811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:12:28.368438 6811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:12:28.368535 6811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:12:28.368625 6811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:12:28.375880 6811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.75.212:43649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:12:28.377060 6811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:12:28.378146 6811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:12:28.387989 6819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:12:28.388121 6811 server_base.cc:1034] running on GCE node
W20260923 18:12:28.388299 6821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:12:28.388557 6818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:12:28.389025 6811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:12:28.389858 6811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:12:28.397284 6811 hybrid_clock.cc:648] HybridClock initialized: now 1790187148397233 us; error 60 us; skew 500 ppm
Sep 23 18:12:28 dist-test-slave-wsx5 krb5kdc[6254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:12:28.401455 6811 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:12:28.402706 6811 webserver.cc:460] Webserver started at http://127.27.75.193:43769/ using document root <none> and password file <none>
I20260923 18:12:28.403501 6811 fs_manager.cc:359] Metadata directory not provided
I20260923 18:12:28.403647 6811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:12:28.403916 6811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:12:28 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:12:28.422699 6811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "e208dc90c60f4222b5b0e141a3354ac5"
format_stamp: "Formatted at 2026-09-23 18:12:28 on dist-test-slave-wsx5"
server_key: "b1136e352bc53ebd8e82698029d0a99a"
server_key_iv: "bfd5c1e986f3628ef0a291820aaa49c6"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:28.423573 6811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "e208dc90c60f4222b5b0e141a3354ac5"
format_stamp: "Formatted at 2026-09-23 18:12:28 on dist-test-slave-wsx5"
server_key: "b1136e352bc53ebd8e82698029d0a99a"
server_key_iv: "bfd5c1e986f3628ef0a291820aaa49c6"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:28.428822 6811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:12:28.456681 6830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:12:28.458484 6811 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260923 18:12:28.458747 6811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "e208dc90c60f4222b5b0e141a3354ac5"
format_stamp: "Formatted at 2026-09-23 18:12:28 on dist-test-slave-wsx5"
server_key: "b1136e352bc53ebd8e82698029d0a99a"
server_key_iv: "bfd5c1e986f3628ef0a291820aaa49c6"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:28.458987 6811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:12:28.476477 6811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:12:28.481246 6811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:12:28.481539 6811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:12:28.482378 6811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:12:28.483601 6811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:12:28.483716 6811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:12:28.483848 6811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:12:28.483932 6811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:12:28.496105 6811 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36001
I20260923 18:12:28.497114 6943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36001 every 8 connection(s)
I20260923 18:12:28.497478 6811 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:12:28.503085 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 6811
I20260923 18:12:28.503211 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:12:28.503526 27951 external_mini_cluster.cc:1428] Setting key 9b39441f01ef1497a4a843aa03fa83b0
Sep 23 18:12:28 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:12:28.516955 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:28.499647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:33349 (local address 127.27.75.254:34941)
0923 18:12:28.499918 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 18:12:28.499922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:12:28.500945 (+ 1023us) server_negotiation.cc:407] Connection header received
0923 18:12:28.502337 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:12:28.502341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:12:28.502408 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:12:28.502516 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:12:28.504346 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:28.505342 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:28.507253 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:28.507625 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:28.507716 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:28.510933 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:12:28.510938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:12:28.510940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:12:28.510983 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:12:28.512893 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:12:28.514062 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:12:28.514066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:12:28.514069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:12:28.514167 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:12:28.515058 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:12:28.515062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:12:28.515064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:12:28.515333 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:12:28.515421 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:12:28.516607 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0923 18:12:28.516752 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:12:28.519111 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:28.500180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34941 (local address 127.27.75.193:33349)
0923 18:12:28.500756 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0923 18:12:28.500827 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 18:12:28.501996 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:12:28.502886 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:12:28.502903 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:12:28.503368 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:12:28.504124 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:28.504150 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:28.505527 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:28.505530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:28.507065 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:28.507080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:28.507965 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:28.507970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:28.508210 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:28.508906 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:12:28.508929 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:12:28.510695 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:12:28.513490 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:12:28.513499 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:12:28.513521 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:12:28.513925 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:12:28.514294 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:12:28.514297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:12:28.514310 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:12:28.514520 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:12:28.515493 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:12:28.515500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:12:28.516248 (+ 748us) client_negotiation.cc:769] Sending connection context
0923 18:12:28.516500 (+ 252us) client_negotiation.cc:240] Negotiation successful
0923 18:12:28.516877 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":166,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:12:28.521032 6944 heartbeater.cc:347] Connected to a master server at 127.27.75.254:34941
I20260923 18:12:28.521350 6944 heartbeater.cc:464] Registering TS with master...
I20260923 18:12:28.522018 6944 heartbeater.cc:511] Master 127.27.75.254:34941 requested a full tablet report, sending...
I20260923 18:12:28.525146 6730 ts_manager.cc:194] Registered new tserver with Master: e208dc90c60f4222b5b0e141a3354ac5 (127.27.75.193:36001)
I20260923 18:12:28.526837 6730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:33349
Sep 23 18:12:28 dist-test-slave-wsx5 krb5kdc[6254](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790187148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:12:28.548734 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:43649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34941
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:12:28.703552 6950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:12:28.703953 6950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:12:28.704021 6950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:12:28.710389 6950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:12:28.710479 6950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:12:28.710572 6950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:12:28.718961 6950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.75.212:43649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:12:28.720407 6950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:12:28.721784 6950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:12:28.735497 6957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:12:28.736109 6956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:12:28.736234 6959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:12:28.737578 6950 server_base.cc:1034] running on GCE node
I20260923 18:12:28.738153 6950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:12:28.738994 6950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:12:28.740338 6950 hybrid_clock.cc:648] HybridClock initialized: now 1790187148740304 us; error 51 us; skew 500 ppm
Sep 23 18:12:28 dist-test-slave-wsx5 krb5kdc[6254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:12:28.744349 6950 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:12:28.745687 6950 webserver.cc:460] Webserver started at http://127.27.75.194:41081/ using document root <none> and password file <none>
I20260923 18:12:28.746470 6950 fs_manager.cc:359] Metadata directory not provided
I20260923 18:12:28.746583 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:12:28.746855 6950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:12:28 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:12:28.760560 6950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "881e4a5af6ae4ec09ad0df5df3d525d3"
format_stamp: "Formatted at 2026-09-23 18:12:28 on dist-test-slave-wsx5"
server_key: "345b3e971c0464c55dd9a2311c9c635b"
server_key_iv: "5313a2dae3791e677a537d72b417b763"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:28.761401 6950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "881e4a5af6ae4ec09ad0df5df3d525d3"
format_stamp: "Formatted at 2026-09-23 18:12:28 on dist-test-slave-wsx5"
server_key: "345b3e971c0464c55dd9a2311c9c635b"
server_key_iv: "5313a2dae3791e677a537d72b417b763"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:28.767341 6950 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260923 18:12:28.784082 6966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:12:28.785979 6950 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:12:28.786176 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "881e4a5af6ae4ec09ad0df5df3d525d3"
format_stamp: "Formatted at 2026-09-23 18:12:28 on dist-test-slave-wsx5"
server_key: "345b3e971c0464c55dd9a2311c9c635b"
server_key_iv: "5313a2dae3791e677a537d72b417b763"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:28.786339 6950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:12:28.801584 6950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:12:28.804979 6950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:12:28.805183 6950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:12:28.805857 6950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:12:28.806823 6950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:12:28.806886 6950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:12:28.806943 6950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:12:28.806969 6950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:12:28.816991 6950 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:44983
I20260923 18:12:28.818212 6950 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:12:28.823632 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 6950
I20260923 18:12:28.823763 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:12:28.824091 27951 external_mini_cluster.cc:1428] Setting key 1e7114bd362e4eef77f3881b36b64971
I20260923 18:12:28.830544 7079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:44983 every 8 connection(s)
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
Sep 23 18:12:28 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:12:28.851660 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:28.834812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:44839 (local address 127.27.75.254:34941)
0923 18:12:28.834975 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 18:12:28.834980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:12:28.837706 (+ 2726us) server_negotiation.cc:407] Connection header received
0923 18:12:28.839101 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:12:28.839105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:12:28.839170 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:12:28.839229 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:12:28.841013 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:28.841872 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:28.843587 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:28.843919 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:28.843999 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:28.847278 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:12:28.847282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:12:28.847285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:12:28.847318 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:12:28.849039 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:12:28.849727 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:12:28.849730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:12:28.849732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:12:28.849823 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:12:28.850271 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:12:28.850275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:12:28.850276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:12:28.850510 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:12:28.850594 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 18:12:28.851343 (+ 749us) server_negotiation.cc:299] Negotiation successful
0923 18:12:28.851484 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 18:12:28.853498 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:28.833571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34941 (local address 127.27.75.194:44839)
0923 18:12:28.837508 (+ 3937us) negotiation.cc:107] Waiting for socket to connect
0923 18:12:28.837581 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 18:12:28.838802 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:12:28.839681 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:12:28.839692 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:12:28.840087 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:12:28.840804 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:28.840826 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:28.842033 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:28.842036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:28.843449 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:28.843461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:28.844218 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:28.844222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:28.844429 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:28.845183 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:12:28.845206 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:12:28.847059 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:12:28.849212 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:12:28.849218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:12:28.849237 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:12:28.849608 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:12:28.849955 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:12:28.849958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:12:28.849960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:12:28.850153 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:12:28.850661 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:12:28.850667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:12:28.851029 (+ 362us) client_negotiation.cc:769] Sending connection context
0923 18:12:28.851933 (+ 904us) client_negotiation.cc:240] Negotiation successful
0923 18:12:28.852339 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3665,"thread_start_us":197,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
I20260923 18:12:28.854918 7080 heartbeater.cc:347] Connected to a master server at 127.27.75.254:34941
I20260923 18:12:28.855181 7080 heartbeater.cc:464] Registering TS with master...
I20260923 18:12:28.855816 7080 heartbeater.cc:511] Master 127.27.75.254:34941 requested a full tablet report, sending...
I20260923 18:12:28.857034 6730 ts_manager.cc:194] Registered new tserver with Master: 881e4a5af6ae4ec09ad0df5df3d525d3 (127.27.75.194:44983)
I20260923 18:12:28.857836 6730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:44839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:12:28.877015 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:43649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34941
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:12:29 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
W20260923 18:12:29.114446 7085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:12:29.114835 7085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:12:29.114894 7085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:12:29.120905 7085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:12:29.121007 7085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:12:29.121094 7085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:12:29.129299 7085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.75.212:43649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:34941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:12:29.130754 7085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:12:29.132143 7085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:12:29.155277 7093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:12:29.161494 7091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:12:29.161705 7090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:12:29.165853 7085 server_base.cc:1034] running on GCE node
I20260923 18:12:29.166481 7085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:12:29.167373 7085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:12:29.183590 7085 hybrid_clock.cc:648] HybridClock initialized: now 1790187149183468 us; error 145 us; skew 500 ppm
Sep 23 18:12:29 dist-test-slave-wsx5 krb5kdc[6254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:12:29.194170 7085 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:12:29.195562 7085 webserver.cc:460] Webserver started at http://127.27.75.195:44691/ using document root <none> and password file <none>
I20260923 18:12:29.196580 7085 fs_manager.cc:359] Metadata directory not provided
I20260923 18:12:29.196751 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:12:29.197145 7085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:12:29 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:12:29.211558 7085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "eb1c7e82e91546d082b9cd96ca3bdb81"
format_stamp: "Formatted at 2026-09-23 18:12:29 on dist-test-slave-wsx5"
server_key: "ebcd3a79e90d16fad2f8825b9debc394"
server_key_iv: "78774d73d9b5ea4d3d6b0641bf90f084"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:29.212137 7085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "eb1c7e82e91546d082b9cd96ca3bdb81"
format_stamp: "Formatted at 2026-09-23 18:12:29 on dist-test-slave-wsx5"
server_key: "ebcd3a79e90d16fad2f8825b9debc394"
server_key_iv: "78774d73d9b5ea4d3d6b0641bf90f084"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:29.225407 7085 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260923 18:12:29.244989 7100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:12:29.246771 7085 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:12:29.246904 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "eb1c7e82e91546d082b9cd96ca3bdb81"
format_stamp: "Formatted at 2026-09-23 18:12:29 on dist-test-slave-wsx5"
server_key: "ebcd3a79e90d16fad2f8825b9debc394"
server_key_iv: "78774d73d9b5ea4d3d6b0641bf90f084"
server_key_version: "kuduclusterkey@0"
I20260923 18:12:29.247037 7085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:12:29.270009 7085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:12:29.273624 7085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:12:29.273867 7085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:12:29.275091 7085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:12:29.276249 7085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:12:29.276376 7085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:12:29.276476 7085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:12:29.276536 7085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:12:29.286665 7085 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:34773
I20260923 18:12:29.287880 7085 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:12:29.288897 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 7085
I20260923 18:12:29.289031 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:12:29.289368 27951 external_mini_cluster.cc:1428] Setting key c1e71053c3273cd0f8d2a871b7c1e9be
I20260923 18:12:29.289995 7213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:34773 every 8 connection(s)
Sep 23 18:12:29 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:12:29.309710 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.293716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:34193 (local address 127.27.75.254:34941)
0923 18:12:29.293838 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 18:12:29.293842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:12:29.294210 (+ 368us) server_negotiation.cc:407] Connection header received
0923 18:12:29.295140 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:12:29.295143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:12:29.295198 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:12:29.295270 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:12:29.296897 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.297720 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.299507 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.299851 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.299929 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.303177 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:12:29.303181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:12:29.303184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:12:29.303217 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:12:29.306893 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:12:29.307672 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:12:29.307676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:12:29.307678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:12:29.307781 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:12:29.308234 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:12:29.308238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:12:29.308239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:12:29.308506 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:12:29.308586 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 18:12:29.309388 (+ 802us) server_negotiation.cc:299] Negotiation successful
0923 18:12:29.309536 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:12:29.311759 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.292486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:34941 (local address 127.27.75.195:34193)
0923 18:12:29.294051 (+ 1565us) negotiation.cc:107] Waiting for socket to connect
0923 18:12:29.294112 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 18:12:29.294922 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:12:29.295588 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:12:29.295597 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:12:29.295975 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:12:29.296671 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.296692 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.297883 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.297886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.299291 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.299303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.300151 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.300155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.300375 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.301106 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:12:29.301155 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:12:29.302955 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:12:29.307073 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:12:29.307082 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:12:29.307105 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:12:29.307551 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:12:29.307901 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:12:29.307904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:12:29.307906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:12:29.308122 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:12:29.308674 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:12:29.308681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:12:29.309057 (+ 376us) client_negotiation.cc:769] Sending connection context
0923 18:12:29.310196 (+ 1139us) client_negotiation.cc:240] Negotiation successful
0923 18:12:29.310604 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"thread_start_us":170,"threads_started":1}
I20260923 18:12:29.313431 7214 heartbeater.cc:347] Connected to a master server at 127.27.75.254:34941
I20260923 18:12:29.313687 7214 heartbeater.cc:464] Registering TS with master...
I20260923 18:12:29.314278 7214 heartbeater.cc:511] Master 127.27.75.254:34941 requested a full tablet report, sending...
I20260923 18:12:29.315578 6730 ts_manager.cc:194] Registered new tserver with Master: eb1c7e82e91546d082b9cd96ca3bdb81 (127.27.75.195:34773)
I20260923 18:12:29.316339 6730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:34193
I20260923 18:12:29.320962 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:12:29.337997 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.323881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.27.75.254:34941)
0923 18:12:29.324007 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 18:12:29.324012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:12:29.324635 (+ 623us) server_negotiation.cc:407] Connection header received
0923 18:12:29.324831 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:12:29.324835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:12:29.324891 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:12:29.324945 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:12:29.325984 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.326791 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.331176 (+ 4385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.331576 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.331677 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.333819 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:12:29.333823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:12:29.333826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:12:29.333863 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:12:29.335555 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:12:29.336224 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:12:29.336228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:12:29.336230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:12:29.336350 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:12:29.336765 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:12:29.336770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:12:29.336772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:12:29.337096 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:12:29.337272 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 18:12:29.337668 (+ 396us) server_negotiation.cc:299] Negotiation successful
0923 18:12:29.337803 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:12:29.342648 6730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:12:29.347675 6730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:12:29.372174 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.359753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:44983 (local address 127.0.0.1:45120)
0923 18:12:29.364865 (+ 5112us) negotiation.cc:107] Waiting for socket to connect
0923 18:12:29.364934 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 18:12:29.365085 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:12:29.366664 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:12:29.366670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:12:29.366693 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:12:29.366868 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.366875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.368118 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.368122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.369514 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.369523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.371617 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.371620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.371752 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.371800 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 18:12:29.371892 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 18:12:29.372014 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4904,"thread_start_us":122,"threads_started":1}
I20260923 18:12:29.373047 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.361305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.27.75.194:44983)
0923 18:12:29.365796 (+ 4491us) server_negotiation.cc:206] Beginning negotiation
0923 18:12:29.365802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:12:29.365826 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:12:29.365910 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:12:29.365915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:12:29.366180 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:12:29.367029 (+ 849us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:12:29.367187 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.367998 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.369668 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.371501 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.372564 (+ 1063us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.372743 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 18:12:29.372809 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 18:12:29.372909 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4172,"thread_start_us":65,"threads_started":1}
I20260923 18:12:29.376149 7014 tablet_service.cc:1467] Processing CreateTablet for tablet bb8afcf9009e4ee58cd052e59ac85921 (DEFAULT_TABLE table=test-table [id=dc488a6a9a1a48128fa437ed588ce3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:12:29.377215 7014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8afcf9009e4ee58cd052e59ac85921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:12:29.382099 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.362460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36001 (local address 127.0.0.1:58872)
0923 18:12:29.366424 (+ 3964us) negotiation.cc:107] Waiting for socket to connect
0923 18:12:29.366436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:12:29.366520 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:12:29.368918 (+ 2398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:12:29.368921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:12:29.368938 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:12:29.369021 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.369026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.371501 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.371504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.372840 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.372850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.381614 (+ 8764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.381619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.381733 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.381761 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 18:12:29.381847 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 18:12:29.381939 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3902,"thread_start_us":109,"threads_started":1}
I20260923 18:12:29.382925 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.362912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.27.75.193:36001)
0923 18:12:29.368283 (+ 5371us) server_negotiation.cc:206] Beginning negotiation
0923 18:12:29.368290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:12:29.368314 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:12:29.368387 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:12:29.368392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:12:29.368616 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:12:29.368772 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:12:29.370302 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.371365 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.380347 (+ 8982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.381265 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.381378 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.381535 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 18:12:29.382647 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0923 18:12:29.382780 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5256,"thread_start_us":120,"threads_started":1}
I20260923 18:12:29.386433 6878 tablet_service.cc:1467] Processing CreateTablet for tablet bb8afcf9009e4ee58cd052e59ac85921 (DEFAULT_TABLE table=test-table [id=dc488a6a9a1a48128fa437ed588ce3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:12:29.387773 6878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8afcf9009e4ee58cd052e59ac85921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:12:29.388418 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.359799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:34773 (local address 127.0.0.1:34220)
0923 18:12:29.365407 (+ 5608us) negotiation.cc:107] Waiting for socket to connect
0923 18:12:29.365525 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0923 18:12:29.365619 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:12:29.370059 (+ 4440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:12:29.370062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:12:29.370079 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:12:29.370177 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.370183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.384091 (+ 13908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.384095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.386495 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.386508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.387936 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.387940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.388046 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.388074 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 18:12:29.388168 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 18:12:29.388264 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5542,"thread_start_us":97,"threads_started":1}
I20260923 18:12:29.389230 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.363849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.27.75.195:34773)
0923 18:12:29.369431 (+ 5582us) server_negotiation.cc:206] Beginning negotiation
0923 18:12:29.369442 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:12:29.369465 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:12:29.369541 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:12:29.369546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:12:29.369765 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:12:29.369914 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:12:29.372990 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.380171 (+ 7181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.386675 (+ 6504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.387565 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.387688 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.387858 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 18:12:29.388908 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0923 18:12:29.389036 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5470,"thread_start_us":135,"threads_started":1}
I20260923 18:12:29.392737 7148 tablet_service.cc:1467] Processing CreateTablet for tablet bb8afcf9009e4ee58cd052e59ac85921 (DEFAULT_TABLE table=test-table [id=dc488a6a9a1a48128fa437ed588ce3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:12:29.395654 7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8afcf9009e4ee58cd052e59ac85921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:12:29.397202 7232 tablet_bootstrap.cc:492] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3: Bootstrap starting.
I20260923 18:12:29.397200 7233 tablet_bootstrap.cc:492] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5: Bootstrap starting.
I20260923 18:12:29.399971 7233 tablet_bootstrap.cc:654] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5: Neither blocks nor log segments found. Creating new log.
I20260923 18:12:29.400956 7233 log.cc:824] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5: Log is configured to *not* fsync() on all Append() calls
I20260923 18:12:29.403154 7233 tablet_bootstrap.cc:492] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5: No bootstrap required, opened a new log
I20260923 18:12:29.403369 7233 ts_tablet_manager.cc:1397] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260923 18:12:29.404907 7232 tablet_bootstrap.cc:654] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3: Neither blocks nor log segments found. Creating new log.
I20260923 18:12:29.405861 7232 log.cc:824] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3: Log is configured to *not* fsync() on all Append() calls
I20260923 18:12:29.408560 7233 raft_consensus.cc:348] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44983 } } peers { permanent_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34773 } } peers { permanent_uuid: "e208dc90c60f4222b5b0e141a3354ac5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36001 } }
I20260923 18:12:29.409013 7233 raft_consensus.cc:374] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:12:29.409111 7233 raft_consensus.cc:729] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e208dc90c60f4222b5b0e141a3354ac5, State: Initialized, Role: FOLLOWER
I20260923 18:12:29.409878 7233 consensus_queue.cc:260] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44983 } } peers { permanent_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34773 } } peers { permanent_uuid: "e208dc90c60f4222b5b0e141a3354ac5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36001 } }
I20260923 18:12:29.411149 7233 ts_tablet_manager.cc:1428] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260923 18:12:29.411567 6944 heartbeater.cc:503] Master 127.27.75.254:34941 was elected leader, sending a full tablet report...
I20260923 18:12:29.411900 7235 tablet_bootstrap.cc:492] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81: Bootstrap starting.
I20260923 18:12:29.413398 7232 tablet_bootstrap.cc:492] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3: No bootstrap required, opened a new log
I20260923 18:12:29.413635 7232 ts_tablet_manager.cc:1397] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20260923 18:12:29.415546 7235 tablet_bootstrap.cc:654] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81: Neither blocks nor log segments found. Creating new log.
I20260923 18:12:29.416460 7235 log.cc:824] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81: Log is configured to *not* fsync() on all Append() calls
I20260923 18:12:29.418445 7232 raft_consensus.cc:348] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44983 } } peers { permanent_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34773 } } peers { permanent_uuid: "e208dc90c60f4222b5b0e141a3354ac5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36001 } }
I20260923 18:12:29.418733 7232 raft_consensus.cc:374] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:12:29.418725 7235 tablet_bootstrap.cc:492] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81: No bootstrap required, opened a new log
I20260923 18:12:29.418781 7232 raft_consensus.cc:729] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e4a5af6ae4ec09ad0df5df3d525d3, State: Initialized, Role: FOLLOWER
I20260923 18:12:29.418936 7235 ts_tablet_manager.cc:1397] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:12:29.419355 7232 consensus_queue.cc:260] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44983 } } peers { permanent_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34773 } } peers { permanent_uuid: "e208dc90c60f4222b5b0e141a3354ac5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36001 } }
I20260923 18:12:29.420084 7080 heartbeater.cc:503] Master 127.27.75.254:34941 was elected leader, sending a full tablet report...
I20260923 18:12:29.420569 7232 ts_tablet_manager.cc:1428] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260923 18:12:29.422808 7235 raft_consensus.cc:348] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44983 } } peers { permanent_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34773 } } peers { permanent_uuid: "e208dc90c60f4222b5b0e141a3354ac5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36001 } }
I20260923 18:12:29.423875 7235 raft_consensus.cc:374] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:12:29.424017 7235 raft_consensus.cc:729] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1c7e82e91546d082b9cd96ca3bdb81, State: Initialized, Role: FOLLOWER
I20260923 18:12:29.424645 7235 consensus_queue.cc:260] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44983 } } peers { permanent_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34773 } } peers { permanent_uuid: "e208dc90c60f4222b5b0e141a3354ac5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36001 } }
I20260923 18:12:29.425383 7214 heartbeater.cc:503] Master 127.27.75.254:34941 was elected leader, sending a full tablet report...
I20260923 18:12:29.429862 7235 ts_tablet_manager.cc:1428] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
Sep 23 18:12:29 dist-test-slave-wsx5 krb5kdc[6254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:12:29.459613 7240 raft_consensus.cc:482] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:12:29.459908 7240 raft_consensus.cc:504] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44983 } } peers { permanent_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34773 } } peers { permanent_uuid: "e208dc90c60f4222b5b0e141a3354ac5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36001 } }
I20260923 18:12:29.461485 7240 leader_election.cc:288] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb1c7e82e91546d082b9cd96ca3bdb81 (127.27.75.195:34773), e208dc90c60f4222b5b0e141a3354ac5 (127.27.75.193:36001)
I20260923 18:12:29.466792 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.462237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:42641 (local address 127.27.75.193:36001)
0923 18:12:29.462539 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0923 18:12:29.462544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:12:29.462561 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:12:29.462632 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:12:29.462635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:12:29.462683 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:12:29.462766 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:12:29.463114 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.463884 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.465431 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.466001 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.466096 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.466149 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 18:12:29.466566 (+ 417us) server_negotiation.cc:299] Negotiation successful
0923 18:12:29.466654 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260923 18:12:29.467222 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.461752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36001 (local address 127.27.75.194:42641)
0923 18:12:29.462337 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0923 18:12:29.462353 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:12:29.462458 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:12:29.462799 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:12:29.462803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:12:29.462839 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:12:29.462993 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.463000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.464013 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.464018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.465289 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.465300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.466261 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.466264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.466416 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.466440 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:12:29.467001 (+ 561us) client_negotiation.cc:240] Negotiation successful
0923 18:12:29.467088 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":33,"spinlock_wait_cycles":14208,"thread_start_us":135,"threads_started":1}
I20260923 18:12:29.467839 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8afcf9009e4ee58cd052e59ac85921" candidate_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e208dc90c60f4222b5b0e141a3354ac5" is_pre_election: true
I20260923 18:12:29.468135 6898 raft_consensus.cc:2454] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881e4a5af6ae4ec09ad0df5df3d525d3 in term 0.
I20260923 18:12:29.468667 6970 leader_election.cc:302] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e4a5af6ae4ec09ad0df5df3d525d3, e208dc90c60f4222b5b0e141a3354ac5; no voters:
I20260923 18:12:29.468961 7240 raft_consensus.cc:2781] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:12:29.469045 7240 raft_consensus.cc:482] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:12:29.469084 7240 raft_consensus.cc:3037] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:12:29.470202 7240 raft_consensus.cc:504] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44983 } } peers { permanent_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34773 } } peers { permanent_uuid: "e208dc90c60f4222b5b0e141a3354ac5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36001 } }
I20260923 18:12:29.470577 7240 leader_election.cc:288] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [CANDIDATE]: Term 1 election: Requested vote from peers eb1c7e82e91546d082b9cd96ca3bdb81 (127.27.75.195:34773), e208dc90c60f4222b5b0e141a3354ac5 (127.27.75.193:36001)
I20260923 18:12:29.470789 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.462406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:55785 (local address 127.27.75.195:34773)
0923 18:12:29.462937 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 18:12:29.462942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:12:29.465542 (+ 2600us) server_negotiation.cc:407] Connection header received
0923 18:12:29.465777 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:12:29.465782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:12:29.465846 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:12:29.465911 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:12:29.466528 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.467280 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.469469 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.470076 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.470175 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.470226 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 18:12:29.470569 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 18:12:29.470663 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445}
I20260923 18:12:29.471000 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8afcf9009e4ee58cd052e59ac85921" candidate_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e208dc90c60f4222b5b0e141a3354ac5"
I20260923 18:12:29.471145 6898 raft_consensus.cc:3037] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:12:29.471513 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.461768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:34773 (local address 127.27.75.194:55785)
0923 18:12:29.465387 (+ 3619us) negotiation.cc:107] Waiting for socket to connect
0923 18:12:29.465403 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:12:29.465620 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:12:29.466190 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:12:29.466194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:12:29.466218 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:12:29.466396 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.466402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.467690 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.467693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.468864 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:12:29.468876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.470331 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.470334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:12:29.470432 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.470454 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:12:29.471037 (+ 583us) client_negotiation.cc:240] Negotiation successful
0923 18:12:29.471117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3541,"mutex_wait_us":27,"thread_start_us":129,"threads_started":1}
I20260923 18:12:29.472510 6898 raft_consensus.cc:2454] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881e4a5af6ae4ec09ad0df5df3d525d3 in term 1.
I20260923 18:12:29.472932 6970 leader_election.cc:302] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e4a5af6ae4ec09ad0df5df3d525d3, e208dc90c60f4222b5b0e141a3354ac5; no voters:
I20260923 18:12:29.473181 7240 raft_consensus.cc:2781] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:12:29.473477 7240 raft_consensus.cc:686] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 1 LEADER]: Becoming Leader. State: Replica: 881e4a5af6ae4ec09ad0df5df3d525d3, State: Running, Role: LEADER
I20260923 18:12:29.473882 7240 consensus_queue.cc:237] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44983 } } peers { permanent_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34773 } } peers { permanent_uuid: "e208dc90c60f4222b5b0e141a3354ac5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36001 } }
I20260923 18:12:29.476817 7151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8afcf9009e4ee58cd052e59ac85921" candidate_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" is_pre_election: true
I20260923 18:12:29.477160 7151 raft_consensus.cc:2454] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881e4a5af6ae4ec09ad0df5df3d525d3 in term 0.
I20260923 18:12:29.477501 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8afcf9009e4ee58cd052e59ac85921" candidate_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81"
I20260923 18:12:29.477624 7150 raft_consensus.cc:3037] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:12:29.477998 6729 catalog_manager.cc:5495] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 881e4a5af6ae4ec09ad0df5df3d525d3 (127.27.75.194). New cstate: current_term: 1 leader_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e4a5af6ae4ec09ad0df5df3d525d3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e208dc90c60f4222b5b0e141a3354ac5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36001 } health_report { overall_health: UNKNOWN } } }
I20260923 18:12:29.478962 7150 raft_consensus.cc:2454] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881e4a5af6ae4ec09ad0df5df3d525d3 in term 1.
W20260923 18:12:29.500281 6945 tablet.cc:2369] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:12:29.501791 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.496422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.27.75.194:44983)
0923 18:12:29.496542 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 18:12:29.496547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:12:29.496561 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:12:29.496653 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:12:29.496656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:12:29.496723 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:12:29.496829 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:12:29.497262 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.498129 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.499325 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:12:29.499640 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:12:29.499739 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:12:29.500174 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:12:29.500908 (+ 734us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:12:29.501181 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0923 18:12:29.501578 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 18:12:29.501649 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:12:29.511615 7150 raft_consensus.cc:1261] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81 [term 1 FOLLOWER]: Refusing update from remote peer 881e4a5af6ae4ec09ad0df5df3d525d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:12:29.511616 6898 raft_consensus.cc:1261] T bb8afcf9009e4ee58cd052e59ac85921 P e208dc90c60f4222b5b0e141a3354ac5 [term 1 FOLLOWER]: Refusing update from remote peer 881e4a5af6ae4ec09ad0df5df3d525d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:12:29.512382 7240 consensus_queue.cc:1035] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e208dc90c60f4222b5b0e141a3354ac5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:12:29.512609 7244 consensus_queue.cc:1035] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1c7e82e91546d082b9cd96ca3bdb81" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:12:29.525841 7250 mvcc.cc:205] Tried to move back new op lower bound from 7332606564391026688 to 7332606564248432640. Current Snapshot: MvccSnapshot[applied={T|T < 7332606564391026688}]
I20260923 18:12:29.544651 6730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:12:29.545015 6730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:12:29.548695 6730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbfad4a0a8d142648f67236125203127: Sending DeleteTablet for 3 replicas of tablet bb8afcf9009e4ee58cd052e59ac85921
I20260923 18:12:29.549788 7014 tablet_service.cc:1514] Processing DeleteTablet for tablet bb8afcf9009e4ee58cd052e59ac85921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:12:29 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:45120
I20260923 18:12:29.550947 7148 tablet_service.cc:1514] Processing DeleteTablet for tablet bb8afcf9009e4ee58cd052e59ac85921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:12:29 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:34220
I20260923 18:12:29.551323 6878 tablet_service.cc:1514] Processing DeleteTablet for tablet bb8afcf9009e4ee58cd052e59ac85921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:12:29 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:58872
I20260923 18:12:29.552600 7259 tablet_replica.cc:330] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3: stopping tablet replica
I20260923 18:12:29.553483 7259 raft_consensus.cc:2229] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:12:29.553462 7257 tablet_replica.cc:330] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81: stopping tablet replica
I20260923 18:12:29.554168 7259 raft_consensus.cc:2258] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:12:29.557109 7259 ts_tablet_manager.cc:1905] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:12:29.557412 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 6811
I20260923 18:12:29.558166 7257 raft_consensus.cc:2229] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:12:29.558576 7257 raft_consensus.cc:2258] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:12:29.562606 7259 ts_tablet_manager.cc:1918] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:12:29.562813 7259 log.cc:1196] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/bb8afcf9009e4ee58cd052e59ac85921
I20260923 18:12:29.563238 7259 ts_tablet_manager.cc:1939] T bb8afcf9009e4ee58cd052e59ac85921 P 881e4a5af6ae4ec09ad0df5df3d525d3: Deleting consensus metadata
I20260923 18:12:29.565021 6717 catalog_manager.cc:4841] TS 881e4a5af6ae4ec09ad0df5df3d525d3 (127.27.75.194:44983): tablet bb8afcf9009e4ee58cd052e59ac85921 (table test-table [id=dc488a6a9a1a48128fa437ed588ce3f3]) successfully deleted
I20260923 18:12:29.567196 7257 ts_tablet_manager.cc:1905] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:12:29.570752 7257 ts_tablet_manager.cc:1918] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:12:29.570891 7257 log.cc:1196] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/bb8afcf9009e4ee58cd052e59ac85921
I20260923 18:12:29.576550 7257 ts_tablet_manager.cc:1939] T bb8afcf9009e4ee58cd052e59ac85921 P eb1c7e82e91546d082b9cd96ca3bdb81: Deleting consensus metadata
I20260923 18:12:29.578212 6716 catalog_manager.cc:4841] TS eb1c7e82e91546d082b9cd96ca3bdb81 (127.27.75.195:34773): tablet bb8afcf9009e4ee58cd052e59ac85921 (table test-table [id=dc488a6a9a1a48128fa437ed588ce3f3]) successfully deleted
W20260923 18:12:29.578277 6717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:36001 (error 108)
I20260923 18:12:29.579128 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 6950
I20260923 18:12:29.585592 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:12:29.580125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36001 (local address 127.0.0.1:58888)
0923 18:12:29.585318 (+ 5193us) negotiation.cc:107] Waiting for socket to connect
0923 18:12:29.585461 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5054}
W20260923 18:12:29.587172 6717 catalog_manager.cc:4584] TS e208dc90c60f4222b5b0e141a3354ac5 (127.27.75.193:36001): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb8afcf9009e4ee58cd052e59ac85921: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36001: connect: Connection refused (error 111)
I20260923 18:12:29.589891 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 7085
I20260923 18:12:29.600904 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 6698
I20260923 18:12:29.606819 27951 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 18:12:29.941946 27951 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 18:12:29.942072 27951 mini_ranger.cc:66] Stopping Ranger...
I20260923 18:12:30.277729 27951 mini_ranger.cc:68] Stopped Ranger
2026-09-23 18:12:30.277 UTC [6284] LOG: received smart shutdown request
2026-09-23 18:12:30.281 UTC [6284] LOG: background worker "logical replication launcher" (PID 6293) exited with exit code 1
2026-09-23 18:12:30.281 UTC [6288] LOG: shutting down
2026-09-23 18:12:30.479 UTC [6284] LOG: database system is shut down
2026-09-23T18:12:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48329 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:12:30 dist-test-slave-wsx5 krb5kdc[7269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:12:30 dist-test-slave-wsx5 krb5kdc[7269](info): set up 2 sockets
Sep 23 18:12:30 dist-test-slave-wsx5 krb5kdc[7269](info): commencing operation
krb5kdc: starting...
W20260923 18:12:31.523554 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:12:31 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:12:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:12:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/rangeradmin_127.27.75.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/rangeradmin_127.27.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/rangerlookup_127.27.75.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/rangerlookup_127.27.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/HTTP_127.27.75.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/HTTP_127.27.75.212@KRBTEST.COM.keytab.
I20260923 18:12:31.711535 27951 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknO4ss8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/postgres -l logfile start
2026-09-23 18:12:34.369 UTC [7299] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 18:12:34.369 UTC [7299] LOG: listening on IPv4 address "127.27.75.212", port 54415
2026-09-23 18:12:34.373 UTC [7299] LOG: listening on Unix socket "/tmp/.s.PGSQL.54415"
2026-09-23 18:12:34.401 UTC [7302] LOG: database system was shut down at 2026-09-23 18:12:32 UTC
2026-09-23 18:12:34.405 UTC [7299] LOG: database system is ready to accept connections
I20260923 18:12:35.279018 27951 mini_postgres.cc:96] Postgres bound to 54415
2026-09-23 18:12:35.284 UTC [7310] FATAL: database "slave" does not exist
127.27.75.212:54415 - accepting connections
I20260923 18:12:35.285470 27951 mini_ranger.cc:161] Starting Ranger...
I20260923 18:12:35.300277 27951 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 18:12:35.836016 27951 mini_ranger.cc:87] Created ranger Postgres database
I20260923 18:12:35.836088 27951 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-admin
2026-09-23 18:12:36,119 [I] DB FLAVOR :POSTGRES
2026-09-23 18:12:36,119 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 18:12:36,119 [I] Checking connection..
2026-09-23 18:12:36,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 18:12:36,378 [I] Checking connection passed.
2026-09-23 18:12:36,378 [I] --------- Verifying version history table ---------
2026-09-23 18:12:36,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:12:36,643 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 18:12:36,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:12:36,874 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 18:12:36,875 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 18:12:36,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 18:12:37.134 UTC [7374] WARNING: there is no transaction in progress
2026-09-23 18:12:37,139 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 18:12:37,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 18:12:37,391 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 18:12:37,391 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 18:12:37,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 18:12:37,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 18:12:37,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wsx5.c.gcp-upstream.internal','N') ;"
2026-09-23 18:12:38,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 18:12:38,400 [I] Table x_portal_user does not exist in database ranger
2026-09-23 18:12:38,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 18:12:38,638 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 18:12:38,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 18:12:38,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 18:12:39,153 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 18:12:39,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 18:12:40.232 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.250 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.268 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.285 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.302 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.322 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.386 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.394 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.412 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.424 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.437 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.454 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.466 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.474 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.482 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:40.491 UTC [7482] WARNING: there is no transaction in progress
2026-09-23 18:12:41,258 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 18:12:41,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 18:12:41,498 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 18:12:41,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 18:12:41,770 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 18:12:41,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 18:12:42,041 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 18:12:42,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 18:12:42,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 18:12:42,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wsx5.c.gcp-upstream.internal';"
2026-09-23 18:12:42,795 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 18:12:42,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 18:12:43,048 [I] DB_PATCHES have already been applied
I20260923 18:12:43.053905 27951 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknO4ss8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 18:12:43.053975 27951 mini_ranger.cc:193] Using host: 127.27.75.212
I20260923 18:12:43.057350 27951 mini_ranger.cc:227] Ranger admin URL: http://127.27.75.212:60883
Sep 23, 2026 6:12:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 6:12:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 6:12:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 6:12:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 6:12:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 6:12:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 6:12:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 18:12:44.070221 27951 mini_ranger.cc:160] Time spent starting Ranger: real 8.785s user 0.000s sys 0.009s
Sep 23, 2026 6:13:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 6:13:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 6:13:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 6:13:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 6:13:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 18:13:07.286532 27951 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/rangerkms_127.27.75.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/rangerkms_127.27.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.75.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.75.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.75.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/HTTP_127.27.75.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.75.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/HTTP_127.27.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 18:13:07 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187187, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 18:13:07.395406 27951 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 18:13:07.406147 27951 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 18:13:08.458222 27951 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 18:13:08.458303 27951 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-kms
I20260923 18:13:08.458321 27951 mini_ranger_kms.cc:227] Using postgres at 127.27.75.212:54415
2026-09-23 18:13:08,622 [I] DB FLAVOR :POSTGRES
2026-09-23 18:13:08,622 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 18:13:08,623 [I] Checking connection
2026-09-23 18:13:08,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 18:13:08,870 [I] connection success
2026-09-23 18:13:08,871 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 18:13:08,871 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 18:13:08,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 18:13:09,134 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 18:13:09,135 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 18:13:09,135 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 18:13:09,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.75.212:54415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 18:13:09,404 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 18:13:09.946058 27951 mini_ranger_kms.cc:310] Created kms service
I20260923 18:13:10.168850 27951 mini_ranger_kms.cc:326] Created kudu user
I20260923 18:13:10.214668 27951 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 18:13:10 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187187, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:13:10.658267 27951 mini_ranger_kms.cc:384] Added ranger policy
I20260923 18:13:10.658423 27951 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknO4ss8/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknO4ss8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 18:13:10.658443 27951 mini_ranger_kms.cc:240] Using host: 127.27.75.212
I20260923 18:13:10.663142 27951 mini_ranger_kms.cc:276] Ranger KMS PID: 7663
I20260923 18:13:10.663230 27951 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.75.212:54171
Sep 23, 2026 6:13:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 6:13:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 6:13:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 6:13:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 6:13:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 6:13:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 18:13:12.732362 27951 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.337s user 0.009s sys 0.140s
I20260923 18:13:12.732501 27951 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 18:13:12.732591 27951 mini_ranger_kms.cc:401] 127.27.75.212:54171/kms/v1/keys
Sep 23 18:13:13 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790187193, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.75.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 18:13:13 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187193, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.75.212@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
Sep 23 18:13:13 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187193, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.75.212@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
Sep 23 18:13:16 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:13:16.225081 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38253
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:38253
--ranger_config_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:54171/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:16.339771 7713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:16.340065 7713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:16.340113 7713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:16.344257 7713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:13:16.344341 7713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:16.344360 7713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:16.344375 7713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:13:16.344389 7713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:13:16.350311 7713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.75.212:54171/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:38253
--ranger_config_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:16.351459 7713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:16.352502 7713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:13:16.362040 7713 server_base.cc:1034] running on GCE node
W20260923 18:13:16.361975 7721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:16.361951 7718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:16.361992 7719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:16.362828 7713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:16.364032 7713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:16.365221 7713 hybrid_clock.cc:648] HybridClock initialized: now 1790187196365192 us; error 47 us; skew 500 ppm
Sep 23 18:13:16 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187196, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:16.368268 7713 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:13:16.369313 7713 webserver.cc:460] Webserver started at http://127.27.75.254:42267/ using document root <none> and password file <none>
I20260923 18:13:16.369856 7713 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:16.369915 7713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:16.370121 7713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:13:16 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187196, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:13:16.418881 7713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "2c09135e1f2249b3a7445a8406abde8b"
format_stamp: "Formatted at 2026-09-23 18:13:16 on dist-test-slave-wsx5"
server_key: "b5f2a02695c0dd57730d403102e9df84"
server_key_iv: "d17e01b15d8c2ad9be78c6a7e0ecb9fa"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:16.419512 7713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "2c09135e1f2249b3a7445a8406abde8b"
format_stamp: "Formatted at 2026-09-23 18:13:16 on dist-test-slave-wsx5"
server_key: "b5f2a02695c0dd57730d403102e9df84"
server_key_iv: "d17e01b15d8c2ad9be78c6a7e0ecb9fa"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:16.423512 7713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:13:16.438755 7728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:16.440197 7713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:13:16.440363 7713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "2c09135e1f2249b3a7445a8406abde8b"
format_stamp: "Formatted at 2026-09-23 18:13:16 on dist-test-slave-wsx5"
server_key: "b5f2a02695c0dd57730d403102e9df84"
server_key_iv: "d17e01b15d8c2ad9be78c6a7e0ecb9fa"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:16.440498 7713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:16.467173 7713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:16.470233 7713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:16.470428 7713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:16.478440 7713 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:38253
I20260923 18:13:16.478459 7780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:38253 every 8 connection(s)
I20260923 18:13:16.479627 7713 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:13:16.482647 7781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:16.483628 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 7713
I20260923 18:13:16.483764 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:13:16.484099 27951 external_mini_cluster.cc:1428] Setting key 9fd88a0cbfeaf77d59276a1b28c3f5ae
I20260923 18:13:16.489861 7781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b: Bootstrap starting.
I20260923 18:13:16.492511 7781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:16.493570 7781 log.cc:824] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:13:16 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:16.496078 7781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b: No bootstrap required, opened a new log
I20260923 18:13:16.499666 7781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } }
I20260923 18:13:16.499955 7781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:16.499998 7781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c09135e1f2249b3a7445a8406abde8b, State: Initialized, Role: FOLLOWER
I20260923 18:13:16.500490 7781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } }
I20260923 18:13:16.500679 7781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:13:16.500783 7781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:13:16.500895 7781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:16.501945 7781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } }
I20260923 18:13:16.502017 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:16.485773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.27.75.254:38253)
0923 18:13:16.486283 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:16.486305 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:16.486349 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:13:16.487228 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:16.487255 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:16.487708 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:16.488103 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:16.489011 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:16.490512 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:16.491855 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:16.492552 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:16.493237 (+ 685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:16.495766 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:16.495787 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:16.495806 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:16.495840 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:16.498662 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:16.499284 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:16.499291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:16.499298 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:16.499449 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:16.499795 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:16.499797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:16.499799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:16.500297 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:16.500496 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:16.500888 (+ 392us) server_negotiation.cc:299] Negotiation successful
0923 18:13:16.501244 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":145,"threads_started":1}
I20260923 18:13:16.502306 7781 leader_election.cc:302] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c09135e1f2249b3a7445a8406abde8b; no voters:
I20260923 18:13:16.502645 7781 leader_election.cc:288] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:13:16.502767 7786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:13:16.502929 7786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 1 LEADER]: Becoming Leader. State: Replica: 2c09135e1f2249b3a7445a8406abde8b, State: Running, Role: LEADER
I20260923 18:13:16.503978 7781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:13:16.504452 7786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } }
I20260923 18:13:16.505798 7788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c09135e1f2249b3a7445a8406abde8b. Latest consensus state: current_term: 1 leader_uuid: "2c09135e1f2249b3a7445a8406abde8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } } }
I20260923 18:13:16.505929 7788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:16.506213 7787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c09135e1f2249b3a7445a8406abde8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } } }
I20260923 18:13:16.506286 7781 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 18:13:16.506312 7787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:16.506699 7800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:13:16.512403 7800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:13:16.519372 7800 catalog_manager.cc:1347] Generated new cluster ID: 21b8d64d57684855b0e9db93c1b1ff94
I20260923 18:13:16.519441 7800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:13:16.529605 7800 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:13:16.530944 7800 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:13:16.551338 7800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b: Generated new TSK 0
I20260923 18:13:16.552285 7800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:13:17.281896 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:54171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38253
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:17.404572 7826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:17.404878 7826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:17.404922 7826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:17.411005 7826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:17.411089 7826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:17.411177 7826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:13:17.418408 7826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.75.212:54171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:17.419600 7826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:17.420647 7826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:17.433100 7836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:17.433228 7833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:17.433100 7834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:17.433665 7826 server_base.cc:1034] running on GCE node
I20260923 18:13:17.434151 7826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:17.434917 7826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:17.436127 7826 hybrid_clock.cc:648] HybridClock initialized: now 1790187197436101 us; error 42 us; skew 500 ppm
Sep 23 18:13:17 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187197, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:17.439114 7826 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:13:17.440191 7826 webserver.cc:460] Webserver started at http://127.27.75.193:46755/ using document root <none> and password file <none>
I20260923 18:13:17.440795 7826 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:17.440866 7826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:17.441084 7826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:13:17 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187197, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:13:17.461711 7826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "1539d7e140c545c882d1c70028fbc9f5"
format_stamp: "Formatted at 2026-09-23 18:13:17 on dist-test-slave-wsx5"
server_key: "3f6341af7885de28d11174c342d034ff"
server_key_iv: "e9151326ff8074a0055332677a6f7f50"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:17.462512 7826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "1539d7e140c545c882d1c70028fbc9f5"
format_stamp: "Formatted at 2026-09-23 18:13:17 on dist-test-slave-wsx5"
server_key: "3f6341af7885de28d11174c342d034ff"
server_key_iv: "e9151326ff8074a0055332677a6f7f50"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:17.474669 7826 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.000s
I20260923 18:13:17.500758 7843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:17.502628 7826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:13:17.502780 7826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "1539d7e140c545c882d1c70028fbc9f5"
format_stamp: "Formatted at 2026-09-23 18:13:17 on dist-test-slave-wsx5"
server_key: "3f6341af7885de28d11174c342d034ff"
server_key_iv: "e9151326ff8074a0055332677a6f7f50"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:17.502912 7826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:17.533074 7826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:17.536950 7826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:17.537226 7826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:17.537992 7826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:17.539098 7826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:17.539191 7826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:17.539279 7826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:17.539315 7826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:17.550186 7826 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36777
I20260923 18:13:17.550228 7958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36777 every 8 connection(s)
I20260923 18:13:17.551307 7826 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:13:17.558938 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 7826
I20260923 18:13:17.559096 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:13:17.559414 27951 external_mini_cluster.cc:1428] Setting key 15496b8552aff402fb3b5ee968fa1ed5
Sep 23 18:13:17 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187197, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:13:17.576913 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:17.554124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38253 (local address 127.27.75.193:44793)
0923 18:13:17.558971 (+ 4847us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:17.559059 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:17.560545 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:17.561647 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:17.561659 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:17.562056 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:17.562965 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:17.562987 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:17.564624 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:17.564628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:17.566507 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:17.566533 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:17.567456 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:17.567462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:17.567849 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:17.568611 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:17.568636 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:17.570453 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:17.572932 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:17.572941 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:17.573034 (+ 93us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:17.573509 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:17.573918 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:17.573922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:17.573924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:17.574144 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:17.574680 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:17.574688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:17.575067 (+ 379us) client_negotiation.cc:769] Sending connection context
0923 18:13:17.575361 (+ 294us) client_negotiation.cc:240] Negotiation successful
0923 18:13:17.575770 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4578,"thread_start_us":107,"threads_started":1}
I20260923 18:13:17.577924 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:17.553660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:44793 (local address 127.27.75.254:38253)
0923 18:13:17.559373 (+ 5713us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:17.559379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:17.559396 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:13:17.561081 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:17.561085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:17.561177 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:17.561263 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:17.563187 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:17.564413 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:17.566726 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:17.567093 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:17.567210 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:17.570737 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:17.570742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:17.570745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:17.570782 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:17.572745 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:17.573680 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:17.573685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:17.573687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:17.573795 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:17.574271 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:17.574274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:17.574276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:17.574555 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:17.574669 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:17.577578 (+ 2909us) server_negotiation.cc:299] Negotiation successful
0923 18:13:17.577748 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5629,"thread_start_us":96,"threads_started":1}
I20260923 18:13:17.579434 7959 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38253
I20260923 18:13:17.579721 7959 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:17.580360 7959 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
I20260923 18:13:17.582695 7745 ts_manager.cc:194] Registered new tserver with Master: 1539d7e140c545c882d1c70028fbc9f5 (127.27.75.193:36777)
I20260923 18:13:17.584286 7745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:44793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:13:17.600839 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:54171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38253
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:13:17 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790187197, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 18:13:17.761425 7965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:17.761850 7965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:17.761922 7965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:17.768277 7965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:17.768474 7965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:17.768616 7965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:13:17.777740 7965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.75.212:54171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:17.779276 7965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:17.780735 7965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:17.796279 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:17.797015 7971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:17.798422 7965 server_base.cc:1034] running on GCE node
W20260923 18:13:17.798882 7972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:17.799974 7965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:17.801026 7965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:17.802246 7965 hybrid_clock.cc:648] HybridClock initialized: now 1790187197802202 us; error 67 us; skew 500 ppm
Sep 23 18:13:17 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187197, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:17.806533 7965 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:13:17.807775 7965 webserver.cc:460] Webserver started at http://127.27.75.194:46681/ using document root <none> and password file <none>
I20260923 18:13:17.808542 7965 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:17.808640 7965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:17.809044 7965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:13:17 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187197, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:13:17.829254 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f"
format_stamp: "Formatted at 2026-09-23 18:13:17 on dist-test-slave-wsx5"
server_key: "d1e7e316db3d43fa2a588c83e86574de"
server_key_iv: "527b122dc19facb6ba215ae89cd8e15b"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:17.830204 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f"
format_stamp: "Formatted at 2026-09-23 18:13:17 on dist-test-slave-wsx5"
server_key: "d1e7e316db3d43fa2a588c83e86574de"
server_key_iv: "527b122dc19facb6ba215ae89cd8e15b"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:17.836230 7965 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260923 18:13:17.867223 7981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:17.869167 7965 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:13:17.869318 7965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f"
format_stamp: "Formatted at 2026-09-23 18:13:17 on dist-test-slave-wsx5"
server_key: "d1e7e316db3d43fa2a588c83e86574de"
server_key_iv: "527b122dc19facb6ba215ae89cd8e15b"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:17.869464 7965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:17.888736 7965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:17.892529 7965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:17.892730 7965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:17.893575 7965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:17.894651 7965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:17.894703 7965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:17.894757 7965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:17.894775 7965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:17.908851 7965 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:43325
I20260923 18:13:17.910284 7965 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:13:17.910622 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 7965
I20260923 18:13:17.910758 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:13:17.911054 27951 external_mini_cluster.cc:1428] Setting key fbcdc93cf11769d00072a6a9c24f5ef4
I20260923 18:13:17.911162 8094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:43325 every 8 connection(s)
Sep 23 18:13:17 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187197, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:17.932262 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:17.916439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:47935 (local address 127.27.75.254:38253)
0923 18:13:17.916573 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:17.916578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:17.917796 (+ 1218us) server_negotiation.cc:407] Connection header received
0923 18:13:17.919579 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:17.919583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:17.919646 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:17.919705 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:17.921566 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:17.922327 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:17.924088 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:17.924450 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:17.924538 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:17.927695 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:17.927699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:17.927702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:17.927734 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:17.929560 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:17.930285 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:17.930290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:17.930292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:17.930389 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:17.930822 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:17.930825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:17.930827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:17.931076 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:17.931165 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:17.931911 (+ 746us) server_negotiation.cc:299] Negotiation successful
0923 18:13:17.932082 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:13:17.934129 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:17.917006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38253 (local address 127.27.75.194:47935)
0923 18:13:17.917628 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:17.917692 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:17.918823 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:17.920238 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:17.920249 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:17.920629 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:17.921385 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:17.921408 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:17.922497 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:17.922501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:17.923916 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:17.923929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:17.924768 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:17.924771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:17.924982 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:17.925710 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:17.925733 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:17.927461 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:17.929720 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:17.929728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:17.929749 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:17.930156 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:17.930500 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:17.930503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:17.930505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:17.930711 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:17.931235 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:17.931242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:17.931607 (+ 365us) client_negotiation.cc:769] Sending connection context
0923 18:13:17.932522 (+ 915us) client_negotiation.cc:240] Negotiation successful
0923 18:13:17.932922 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":161,"threads_started":1}
I20260923 18:13:17.935614 8095 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38253
I20260923 18:13:17.935889 8095 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:17.936653 8095 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:13:17.937943 7745 ts_manager.cc:194] Registered new tserver with Master: 8ba8d54921da4fb6aa243d21a1f2bc6f (127.27.75.194:43325)
I20260923 18:13:17.938748 7745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:47935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:13:17.977415 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:54171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38253
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:13:18 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187197, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
W20260923 18:13:18.228574 8100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:18.228974 8100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:18.229038 8100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:18.235460 8100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:18.235560 8100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:18.235648 8100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:13:18.263252 8100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.75.212:54171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:38253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:18.264743 8100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:18.266230 8100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:18.299983 8106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:18.299957 8105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:18.299968 8108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:18.302620 8100 server_base.cc:1034] running on GCE node
I20260923 18:13:18.303293 8100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:18.304203 8100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:18.319757 8100 hybrid_clock.cc:648] HybridClock initialized: now 1790187198319650 us; error 142 us; skew 500 ppm
Sep 23 18:13:18 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187198, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:18.323741 8100 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:13:18.325008 8100 webserver.cc:460] Webserver started at http://127.27.75.195:43955/ using document root <none> and password file <none>
I20260923 18:13:18.325847 8100 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:18.326004 8100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:18.326321 8100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 18:13:18 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187198, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:13:18.347048 8100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "0e8694cc1d424d71a3ca84af73ba3cf6"
format_stamp: "Formatted at 2026-09-23 18:13:18 on dist-test-slave-wsx5"
server_key: "d197a2e66c339cd8198807d1bab07cc2"
server_key_iv: "54c65fd92bf1986ef6f254624b30533b"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:18.347940 8100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "0e8694cc1d424d71a3ca84af73ba3cf6"
format_stamp: "Formatted at 2026-09-23 18:13:18 on dist-test-slave-wsx5"
server_key: "d197a2e66c339cd8198807d1bab07cc2"
server_key_iv: "54c65fd92bf1986ef6f254624b30533b"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:18.361606 8100 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.004s
I20260923 18:13:18.385068 8115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:18.386839 8100 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260923 18:13:18.387051 8100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "0e8694cc1d424d71a3ca84af73ba3cf6"
format_stamp: "Formatted at 2026-09-23 18:13:18 on dist-test-slave-wsx5"
server_key: "d197a2e66c339cd8198807d1bab07cc2"
server_key_iv: "54c65fd92bf1986ef6f254624b30533b"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:18.387260 8100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:18.412535 8100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:18.416399 8100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:18.416684 8100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:18.417520 8100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:18.418718 8100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:18.418821 8100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:18.418912 8100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:18.418967 8100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:18.429227 8100 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:43199
I20260923 18:13:18.430577 8100 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:13:18.432412 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 8100
I20260923 18:13:18.432549 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:13:18.432873 27951 external_mini_cluster.cc:1428] Setting key fbbd88cc4619b6f233a22dfb909a56e8
I20260923 18:13:18.438138 8228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:43199 every 8 connection(s)
Sep 23 18:13:18 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187198, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:18.457752 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:18.440507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:53285 (local address 127.27.75.254:38253)
0923 18:13:18.442768 (+ 2261us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:18.442773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:18.444569 (+ 1796us) server_negotiation.cc:407] Connection header received
0923 18:13:18.444625 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:18.444628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:18.444701 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:18.444787 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:18.446473 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:18.447274 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:18.449099 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:18.449493 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:18.449579 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:18.452880 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:18.452885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:18.452887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:18.452919 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:18.454886 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:18.455623 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:18.455628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:18.455629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:18.455745 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:18.456206 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:18.456210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:18.456212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:18.456496 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:18.456574 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:18.457388 (+ 814us) server_negotiation.cc:299] Negotiation successful
0923 18:13:18.457546 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2161,"thread_start_us":92,"threads_started":1}
I20260923 18:13:18.459717 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:18.441113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38253 (local address 127.27.75.195:53285)
0923 18:13:18.443065 (+ 1952us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:18.443133 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:18.444231 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:18.445160 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:18.445172 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:18.445566 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:18.446292 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:18.446315 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:18.447442 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:18.447446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:18.448939 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:18.448951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:18.449830 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:18.449835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:18.450059 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:18.450778 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:18.450802 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:18.452627 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:18.455040 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:18.455048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:18.455069 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:18.455478 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:18.455865 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:18.455868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:18.455870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:18.456091 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:18.456671 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:18.456679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:18.457044 (+ 365us) client_negotiation.cc:769] Sending connection context
0923 18:13:18.458161 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0923 18:13:18.458559 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1670,"thread_start_us":525,"threads_started":1}
I20260923 18:13:18.461390 8229 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38253
I20260923 18:13:18.461689 8229 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:18.462343 8229 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
I20260923 18:13:18.463740 7745 ts_manager.cc:194] Registered new tserver with Master: 0e8694cc1d424d71a3ca84af73ba3cf6 (127.27.75.195:43199)
I20260923 18:13:18.464485 7745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:53285
I20260923 18:13:18.471513 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:13:18.472359 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 7826
I20260923 18:13:18.484246 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 7965
I20260923 18:13:18.491614 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 8100
I20260923 18:13:18.499199 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 7713
I20260923 18:13:18.507164 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38253
--webserver_interface=127.27.75.254
--webserver_port=42267
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:38253
--ranger_config_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:54171/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.75.212:54171/kms with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:18.621840 8233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:18.622136 8233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:18.622184 8233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:18.626407 8233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:13:18.626489 8233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:18.626507 8233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:18.626523 8233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:13:18.626536 8233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:13:18.632321 8233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.75.212:54171/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:38253
--ranger_config_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:38253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=42267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:18.633622 8233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:18.634708 8233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:18.644047 8238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:18.644129 8241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:18.644076 8239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:18.644296 8233 server_base.cc:1034] running on GCE node
I20260923 18:13:18.645031 8233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:18.646256 8233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:18.647442 8233 hybrid_clock.cc:648] HybridClock initialized: now 1790187198647431 us; error 35 us; skew 500 ppm
Sep 23 18:13:18 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187198, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:18.650405 8233 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:13:18.651391 8233 webserver.cc:460] Webserver started at http://127.27.75.254:42267/ using document root <none> and password file <none>
I20260923 18:13:18.651947 8233 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:18.652040 8233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 18:13:18 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187198, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:13:18.739544 8233 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:13:18.742348 8249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:18.744037 8233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:13:18.744196 8233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "2c09135e1f2249b3a7445a8406abde8b"
format_stamp: "Formatted at 2026-09-23 18:13:16 on dist-test-slave-wsx5"
server_key: "b5f2a02695c0dd57730d403102e9df84"
server_key_iv: "d17e01b15d8c2ad9be78c6a7e0ecb9fa"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:18.744951 8233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:18.772336 8233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:18.775530 8233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:18.775738 8233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:18.783450 8233 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:38253
I20260923 18:13:18.783507 8301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:38253 every 8 connection(s)
I20260923 18:13:18.784559 8233 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:13:18.786111 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 8233
I20260923 18:13:18.786594 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:54171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:36777
--local_ip_for_outbound_sockets=127.27.75.193
--tserver_master_addrs=127.27.75.254:38253
--webserver_port=46755
--webserver_interface=127.27.75.193
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.75.212:54171/kms with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:13:18.790167 8302 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:13:18.792704 8302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b: Bootstrap starting.
I20260923 18:13:18.815384 8302 log.cc:824] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:18.823902 8302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:13:18.824369 8302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b: Bootstrap complete.
I20260923 18:13:18.829804 8302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } }
I20260923 18:13:18.830366 8302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c09135e1f2249b3a7445a8406abde8b, State: Initialized, Role: FOLLOWER
I20260923 18:13:18.830940 8302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } }
I20260923 18:13:18.831086 8302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:13:18.831163 8302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:13:18.831259 8302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:13:18.833880 8302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } }
I20260923 18:13:18.834271 8302 leader_election.cc:302] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c09135e1f2249b3a7445a8406abde8b; no voters:
I20260923 18:13:18.834615 8302 leader_election.cc:288] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 18:13:18.834751 8306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 2 FOLLOWER]: Leader election won for term 2
I20260923 18:13:18.835037 8306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [term 2 LEADER]: Becoming Leader. State: Replica: 2c09135e1f2249b3a7445a8406abde8b, State: Running, Role: LEADER
I20260923 18:13:18.835382 8306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } }
I20260923 18:13:18.835935 8302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:13:18.836858 8307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2c09135e1f2249b3a7445a8406abde8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } } }
I20260923 18:13:18.836998 8307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:18.837275 8308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c09135e1f2249b3a7445a8406abde8b. Latest consensus state: current_term: 2 leader_uuid: "2c09135e1f2249b3a7445a8406abde8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c09135e1f2249b3a7445a8406abde8b" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 38253 } } }
I20260923 18:13:18.837357 8308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:18.838285 8302 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 18:13:18.838774 8315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:13:18.844090 8315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:13:18.844949 8315 catalog_manager.cc:1259] Loaded cluster ID: 21b8d64d57684855b0e9db93c1b1ff94
I20260923 18:13:18.845014 8315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:13:18.847744 8315 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:13:18.848558 8315 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b: Loaded TSK: 0
I20260923 18:13:18.850188 8315 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:13:18.948263 8304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:18.948607 8304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:18.948680 8304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:18.954695 8304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:18.954779 8304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:18.954867 8304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:13:18.962656 8304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.75.212:54171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:36777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=46755
--tserver_master_addrs=127.27.75.254:38253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:18.964022 8304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:18.965484 8304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:18.978533 8342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:18.979571 8339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:18.979703 8340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:18.979904 8304 server_base.cc:1034] running on GCE node
I20260923 18:13:18.980440 8304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:18.981317 8304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:18.982529 8304 hybrid_clock.cc:648] HybridClock initialized: now 1790187198982504 us; error 48 us; skew 500 ppm
Sep 23 18:13:18 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187198, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:18.987076 8304 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:13:18.988345 8304 webserver.cc:460] Webserver started at http://127.27.75.193:46755/ using document root <none> and password file <none>
I20260923 18:13:18.989086 8304 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:18.989250 8304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 18:13:18 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187198, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:13:19.010404 8304 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:13:19.014109 8350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:19.015805 8304 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 18:13:19.015957 8304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "1539d7e140c545c882d1c70028fbc9f5"
format_stamp: "Formatted at 2026-09-23 18:13:17 on dist-test-slave-wsx5"
server_key: "3f6341af7885de28d11174c342d034ff"
server_key_iv: "e9151326ff8074a0055332677a6f7f50"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:19.016708 8304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:19.043402 8304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:19.047109 8304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:19.047374 8304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:19.048059 8304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:19.049171 8304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:19.049274 8304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:19.049405 8304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:19.049539 8304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:19.062075 8463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36777 every 8 connection(s)
I20260923 18:13:19.062379 8304 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36777
I20260923 18:13:19.063861 8304 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:13:19.065955 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 8304
I20260923 18:13:19.066447 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:54171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:43325
--local_ip_for_outbound_sockets=127.27.75.194
--tserver_master_addrs=127.27.75.254:38253
--webserver_port=46681
--webserver_interface=127.27.75.194
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.75.212:54171/kms with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:13:19 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187198, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:19.084990 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:19.067101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38253 (local address 127.27.75.193:50941)
0923 18:13:19.068403 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:19.068464 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:19.069576 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:19.071491 (+ 1915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:19.071500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:19.071869 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:19.072544 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:19.072566 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.074539 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.074543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:19.075613 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:19.075623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.076418 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.076422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:19.076603 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:19.077210 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:19.077227 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:19.078972 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:19.081547 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:19.081553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:19.081569 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:19.081993 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:19.082371 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:19.082374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:19.082376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:19.082514 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:19.083138 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:19.083144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:19.083393 (+ 249us) client_negotiation.cc:769] Sending connection context
0923 18:13:19.083567 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 18:13:19.083925 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":162,"threads_started":1}
I20260923 18:13:19.084976 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:19.068703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:50941 (local address 127.27.75.254:38253)
0923 18:13:19.069381 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:19.069402 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:19.069451 (+ 49us) server_negotiation.cc:407] Connection header received
0923 18:13:19.070198 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:19.070213 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:19.070629 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:19.070973 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:19.072752 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.074371 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.075759 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.076236 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.076388 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:19.079236 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:19.079243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:19.079262 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:19.079297 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:19.081404 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:19.082132 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:19.082138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:19.082144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:19.082268 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:19.082621 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:19.082624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:19.082626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:19.083013 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:19.083139 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:19.083751 (+ 612us) server_negotiation.cc:299] Negotiation successful
0923 18:13:19.084004 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":121,"threads_started":1}
I20260923 18:13:19.087895 8464 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38253
I20260923 18:13:19.088158 8464 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:19.088681 8464 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
W20260923 18:13:19.091188 8464 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:38253 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 18:13:19.091320 8464 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:19.091517 8464 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
I20260923 18:13:19.092388 8464 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:19.092571 8464 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
W20260923 18:13:19.093351 8464 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:13:19.234550 8467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:19.234884 8467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:19.234948 8467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:19.239521 8467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:19.239593 8467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:19.239678 8467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:13:19.246237 8467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.75.212:54171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:43325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=46681
--tserver_master_addrs=127.27.75.254:38253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:19.247710 8467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:19.249199 8467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:19.262383 8476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:19.262383 8473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:19.262969 8474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:19.263639 8467 server_base.cc:1034] running on GCE node
I20260923 18:13:19.264142 8467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:19.265008 8467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:19.268311 8467 hybrid_clock.cc:648] HybridClock initialized: now 1790187199268222 us; error 115 us; skew 500 ppm
Sep 23 18:13:19 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187199, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:19.276808 8467 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:13:19.278338 8467 webserver.cc:460] Webserver started at http://127.27.75.194:46681/ using document root <none> and password file <none>
I20260923 18:13:19.279150 8467 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:19.279312 8467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 18:13:19 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187199, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:13:19.308182 8467 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:13:19.312929 8484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:19.314929 8467 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260923 18:13:19.315112 8467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f"
format_stamp: "Formatted at 2026-09-23 18:13:17 on dist-test-slave-wsx5"
server_key: "d1e7e316db3d43fa2a588c83e86574de"
server_key_iv: "527b122dc19facb6ba215ae89cd8e15b"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:19.315909 8467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:19.338311 8467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:19.342471 8467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:19.342772 8467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:19.343607 8467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:19.344762 8467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:19.344870 8467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:19.344969 8467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:19.345027 8467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:19.355612 8467 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:43325
I20260923 18:13:19.355700 8598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:43325 every 8 connection(s)
I20260923 18:13:19.357081 8467 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:13:19.357573 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 8467
I20260923 18:13:19.358073 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.75.212:54171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:43199
--local_ip_for_outbound_sockets=127.27.75.195
--tserver_master_addrs=127.27.75.254:38253
--webserver_port=43955
--webserver_interface=127.27.75.195
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.75.212:54171/kms with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:13:19 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187199, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:19.375985 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:19.361298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:60621 (local address 127.27.75.254:38253)
0923 18:13:19.361431 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:19.361436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:19.361455 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:13:19.362216 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:19.362220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:19.362304 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:19.362361 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:19.364500 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.365374 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.367224 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.367576 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.367672 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:19.371006 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:19.371011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:19.371014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:19.371049 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:19.372865 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:19.373660 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:19.373664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:19.373666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:19.373780 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:19.374511 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:19.374514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:19.374516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:19.374782 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:19.374875 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:19.375641 (+ 766us) server_negotiation.cc:299] Negotiation successful
0923 18:13:19.375805 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:13:19.378006 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:19.359996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38253 (local address 127.27.75.194:60621)
0923 18:13:19.360668 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:19.360743 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:19.361920 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:19.363068 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:19.363079 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:19.363466 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:19.364275 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:19.364299 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.365558 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.365561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:19.367050 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:19.367064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.367919 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.367923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:19.368144 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:19.368886 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:19.368909 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:19.370751 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:19.373075 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:19.373082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:19.373103 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:19.373527 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:19.373908 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:19.373912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:19.373914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:19.374373 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:19.374947 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:19.374954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:19.375330 (+ 376us) client_negotiation.cc:769] Sending connection context
0923 18:13:19.376251 (+ 921us) client_negotiation.cc:240] Negotiation successful
0923 18:13:19.376638 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":163,"threads_started":1}
I20260923 18:13:19.379540 8599 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38253
I20260923 18:13:19.379825 8599 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:19.380465 8599 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
W20260923 18:13:19.382370 8599 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:38253 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 18:13:19.382519 8599 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:19.382710 8599 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
I20260923 18:13:19.383625 8599 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:19.383817 8599 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
W20260923 18:13:19.384641 8599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:13:19.496456 8601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:19.496737 8601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:19.496797 8601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:19.501760 8601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:19.501840 8601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:19.501924 8601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:13:19.507859 8601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.75.212:54171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:43199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=43955
--tserver_master_addrs=127.27.75.254:38253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:19.509050 8601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:19.510178 8601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:19.520942 8610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:19.520946 8607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:19.521245 8601 server_base.cc:1034] running on GCE node
W20260923 18:13:19.521070 8608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:19.521740 8601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:19.522434 8601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:19.523638 8601 hybrid_clock.cc:648] HybridClock initialized: now 1790187199523620 us; error 39 us; skew 500 ppm
Sep 23 18:13:19 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187199, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:19.526736 8601 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:13:19.527812 8601 webserver.cc:460] Webserver started at http://127.27.75.195:43955/ using document root <none> and password file <none>
I20260923 18:13:19.528425 8601 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:19.528527 8601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 18:13:19 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187199, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:13:19.561961 8601 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.004s sys 0.000s
I20260923 18:13:19.566872 8618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:19.568769 8601 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260923 18:13:19.569016 8601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "0e8694cc1d424d71a3ca84af73ba3cf6"
format_stamp: "Formatted at 2026-09-23 18:13:18 on dist-test-slave-wsx5"
server_key: "d197a2e66c339cd8198807d1bab07cc2"
server_key_iv: "54c65fd92bf1986ef6f254624b30533b"
server_key_version: "kuduclusterkey@0"
I20260923 18:13:19.569950 8601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:19.599882 8601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:19.603849 8601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:19.604067 8601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:19.604871 8601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:19.606065 8601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:19.606134 8601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:19.606191 8601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:19.606223 8601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:19.618124 8601 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:43199
I20260923 18:13:19.618364 8731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:43199 every 8 connection(s)
I20260923 18:13:19.619539 8601 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:13:19.628340 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 8601
Sep 23 18:13:19 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187199, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:19.637995 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:19.624151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:33587 (local address 127.27.75.254:38253)
0923 18:13:19.624274 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:19.624278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:19.624296 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:13:19.624842 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:19.624846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:19.624907 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:19.625019 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:19.626487 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.627320 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.629761 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.630100 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.630193 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:19.633515 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:19.633520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:19.633523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:19.633556 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:19.635282 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:19.636086 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:19.636089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:19.636091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:19.636183 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:19.636646 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:19.636649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:19.636650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:19.636855 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:19.636965 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:19.637706 (+ 741us) server_negotiation.cc:299] Negotiation successful
0923 18:13:19.637833 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:13:19.639814 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:19.622908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:38253 (local address 127.27.75.195:33587)
0923 18:13:19.623543 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:19.623589 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:19.624588 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:19.625236 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:19.625245 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:19.625623 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:19.626301 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:19.626322 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.627488 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.627492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:19.629602 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:19.629616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.630417 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.630421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:19.630644 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:19.631356 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:19.631379 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:19.633275 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:19.635452 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:19.635458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:19.635479 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:19.635896 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:19.636320 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:19.636323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:19.636325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:19.636543 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:19.636962 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:19.636970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:19.637399 (+ 429us) client_negotiation.cc:769] Sending connection context
0923 18:13:19.638348 (+ 949us) client_negotiation.cc:240] Negotiation successful
0923 18:13:19.638727 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":155,"threads_started":1}
I20260923 18:13:19.641191 8732 heartbeater.cc:347] Connected to a master server at 127.27.75.254:38253
I20260923 18:13:19.641498 8732 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:19.642148 8732 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
I20260923 18:13:19.643808 8266 ts_manager.cc:194] Registered new tserver with Master: 0e8694cc1d424d71a3ca84af73ba3cf6 (127.27.75.195:43199)
I20260923 18:13:19.645469 8266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:33587
I20260923 18:13:19.646219 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:19.629026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.27.75.254:38253)
0923 18:13:19.637323 (+ 8297us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:19.637327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:19.637504 (+ 177us) server_negotiation.cc:407] Connection header received
0923 18:13:19.637654 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:19.637657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:19.637712 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:19.637798 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:19.638689 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.639207 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.640250 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:19.640493 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:19.640583 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:19.642263 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:19.642267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:19.642270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:19.642297 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:19.644080 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:19.644643 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:19.644647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:19.644650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:19.644751 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:19.645211 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:19.645215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:19.645217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:19.645457 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:19.645772 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:19.645877 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 18:13:19.646027 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8205,"thread_start_us":115,"threads_started":1}
Sep 23 18:13:19 dist-test-slave-wsx5 krb5kdc[7269](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790187199, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:20.093622 8464 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:20.093997 8464 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
I20260923 18:13:20.095028 8266 ts_manager.cc:194] Registered new tserver with Master: 1539d7e140c545c882d1c70028fbc9f5 (127.27.75.193:36777)
I20260923 18:13:20.095769 8266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:50941
I20260923 18:13:20.384925 8599 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:20.385289 8599 heartbeater.cc:511] Master 127.27.75.254:38253 requested a full tablet report, sending...
I20260923 18:13:20.386160 8266 ts_manager.cc:194] Registered new tserver with Master: 8ba8d54921da4fb6aa243d21a1f2bc6f (127.27.75.194:43325)
I20260923 18:13:20.386999 8266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:60621
Sep 23 18:13:20 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187199, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:13:20.393520 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:13:20.407234 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.397055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.27.75.254:38253)
0923 18:13:20.397352 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:20.397357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:20.397448 (+ 91us) server_negotiation.cc:407] Connection header received
0923 18:13:20.397613 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:20.397616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:20.397677 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:20.397758 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:20.398874 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.399731 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.400895 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.401281 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.401378 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.403141 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:20.403145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:20.403148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:20.403182 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:20.404943 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:20.405502 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:20.405506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:20.405508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:20.405617 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:20.405898 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:20.405902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:20.405903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:20.406124 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:20.406246 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:20.406916 (+ 670us) server_negotiation.cc:299] Negotiation successful
0923 18:13:20.407053 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":116,"threads_started":1}
I20260923 18:13:20.414091 8266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:13:20.420475 8266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:13:20.453354 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.444483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:43199 (local address 127.0.0.1:51670)
0923 18:13:20.445814 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:20.445871 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:20.446001 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:20.446798 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:20.446804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:20.447003 (+ 199us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:20.447164 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:20.447172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.448772 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.448776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:20.451257 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:20.451268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.452641 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.452645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:20.452766 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.452810 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 18:13:20.452922 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 18:13:20.453190 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"spinlock_wait_cycles":2304,"thread_start_us":106,"threads_started":1}
I20260923 18:13:20.455401 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.445075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.27.75.195:43199)
0923 18:13:20.446307 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:20.446315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:20.446338 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:13:20.446412 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:20.446417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:20.446624 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:20.447327 (+ 703us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:20.447460 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.448617 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.451411 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.452299 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.452405 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.452570 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:20.455126 (+ 2556us) server_negotiation.cc:299] Negotiation successful
0923 18:13:20.455251 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1090,"thread_start_us":169,"threads_started":1}
I20260923 18:13:20.459288 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.445747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:43325 (local address 127.0.0.1:42904)
0923 18:13:20.446734 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:20.446748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:20.448142 (+ 1394us) spinlock_profiling.cc:243] Waited 899 us on lock 0x30423028. stack: 00007f38d6ea4c78 00007f38d6ea51a9 00007f38d6a671d7 00000000005984a6 00000000005984f9 000000000059c488 00007f38d6ea179e 00007f38d6ea0a97 00007f38df602817 00007f38df61507f 00007f38df60b003 00007f38db4f9267 00007f38db4f7819 00007f38db538832 00007f38db53968c 00007f38db54f0d3
0923 18:13:20.448164 (+ 22us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:20.449732 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:20.449735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:20.449753 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:20.449849 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:20.449854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.451305 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.451309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:20.452531 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:20.452542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.458859 (+ 6317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.458863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:20.458963 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.458986 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:13:20.459053 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 18:13:20.459129 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"spinlock_wait_cycles":2067840,"thread_start_us":82,"threads_started":1}
I20260923 18:13:20.459043 8666 tablet_service.cc:1467] Processing CreateTablet for tablet 4071194a3fea42b3b3906d52e8be8096 (DEFAULT_TABLE table=test-table [id=780f663fa66e43288b45735ea3c6d190]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:20.460290 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.446405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36777 (local address 127.0.0.1:54042)
0923 18:13:20.452776 (+ 6371us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:20.452789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:20.452868 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:20.453627 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:20.453630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:20.453658 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:20.453739 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:20.453744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.455183 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.455185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:20.456366 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:20.456376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.459910 (+ 3534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.459914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:20.460007 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.460030 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:13:20.460098 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 18:13:20.460165 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6303,"thread_start_us":84,"threads_started":1}
I20260923 18:13:20.460295 8666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4071194a3fea42b3b3906d52e8be8096. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:20.460723 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.446550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.27.75.193:36777)
0923 18:13:20.448930 (+ 2380us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:20.448937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:20.452995 (+ 4058us) server_negotiation.cc:407] Connection header received
0923 18:13:20.453085 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:20.453090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:20.453339 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:20.453477 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:20.453907 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.455055 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.456502 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.457369 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.457466 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.457602 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:20.460461 (+ 2859us) server_negotiation.cc:299] Negotiation successful
0923 18:13:20.460574 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2256,"thread_start_us":61,"threads_started":1}
I20260923 18:13:20.459798 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.445881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.27.75.194:43325)
0923 18:13:20.449112 (+ 3231us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:20.449145 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:20.449169 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:13:20.449244 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:20.449250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:20.449453 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:20.449587 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:20.449965 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.451162 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.457688 (+ 6526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.458548 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.458668 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.458803 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:20.459543 (+ 740us) server_negotiation.cc:299] Negotiation successful
0923 18:13:20.459663 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3134,"thread_start_us":64,"threads_started":1}
I20260923 18:13:20.464357 8379 tablet_service.cc:1467] Processing CreateTablet for tablet 4071194a3fea42b3b3906d52e8be8096 (DEFAULT_TABLE table=test-table [id=780f663fa66e43288b45735ea3c6d190]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:20.464356 8533 tablet_service.cc:1467] Processing CreateTablet for tablet 4071194a3fea42b3b3906d52e8be8096 (DEFAULT_TABLE table=test-table [id=780f663fa66e43288b45735ea3c6d190]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:20.465517 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4071194a3fea42b3b3906d52e8be8096. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:20.465517 8533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4071194a3fea42b3b3906d52e8be8096. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:20.475358 8766 tablet_bootstrap.cc:492] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5: Bootstrap starting.
I20260923 18:13:20.477911 8766 tablet_bootstrap.cc:654] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:20.478313 8767 tablet_bootstrap.cc:492] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: Bootstrap starting.
I20260923 18:13:20.479048 8766 log.cc:824] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:20.480680 8767 tablet_bootstrap.cc:654] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:20.481554 8767 log.cc:824] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:20.481834 8766 tablet_bootstrap.cc:492] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5: No bootstrap required, opened a new log
I20260923 18:13:20.482036 8766 ts_tablet_manager.cc:1397] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:13:20.483479 8765 tablet_bootstrap.cc:492] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: Bootstrap starting.
I20260923 18:13:20.485992 8765 tablet_bootstrap.cc:654] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:20.486194 8766 raft_consensus.cc:348] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43199 } } peers { permanent_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43325 } } peers { permanent_uuid: "1539d7e140c545c882d1c70028fbc9f5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36777 } }
I20260923 18:13:20.486604 8766 raft_consensus.cc:374] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:20.486706 8766 raft_consensus.cc:729] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1539d7e140c545c882d1c70028fbc9f5, State: Initialized, Role: FOLLOWER
I20260923 18:13:20.486969 8765 log.cc:824] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:20.487342 8766 consensus_queue.cc:260] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43199 } } peers { permanent_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43325 } } peers { permanent_uuid: "1539d7e140c545c882d1c70028fbc9f5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36777 } }
I20260923 18:13:20.488546 8766 ts_tablet_manager.cc:1428] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:13:20.488950 8464 heartbeater.cc:503] Master 127.27.75.254:38253 was elected leader, sending a full tablet report...
I20260923 18:13:20.489936 8767 tablet_bootstrap.cc:492] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: No bootstrap required, opened a new log
I20260923 18:13:20.490152 8767 ts_tablet_manager.cc:1397] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260923 18:13:20.496426 8767 raft_consensus.cc:348] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43199 } } peers { permanent_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43325 } } peers { permanent_uuid: "1539d7e140c545c882d1c70028fbc9f5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36777 } }
I20260923 18:13:20.496881 8767 raft_consensus.cc:374] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:20.496973 8767 raft_consensus.cc:729] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba8d54921da4fb6aa243d21a1f2bc6f, State: Initialized, Role: FOLLOWER
I20260923 18:13:20.497594 8767 consensus_queue.cc:260] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43199 } } peers { permanent_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43325 } } peers { permanent_uuid: "1539d7e140c545c882d1c70028fbc9f5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36777 } }
I20260923 18:13:20.498054 8765 tablet_bootstrap.cc:492] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: No bootstrap required, opened a new log
I20260923 18:13:20.498301 8765 ts_tablet_manager.cc:1397] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260923 18:13:20.498764 8599 heartbeater.cc:503] Master 127.27.75.254:38253 was elected leader, sending a full tablet report...
I20260923 18:13:20.500650 8767 ts_tablet_manager.cc:1428] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260923 18:13:20.505162 8765 raft_consensus.cc:348] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43199 } } peers { permanent_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43325 } } peers { permanent_uuid: "1539d7e140c545c882d1c70028fbc9f5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36777 } }
I20260923 18:13:20.505664 8765 raft_consensus.cc:374] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:20.505771 8765 raft_consensus.cc:729] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e8694cc1d424d71a3ca84af73ba3cf6, State: Initialized, Role: FOLLOWER
I20260923 18:13:20.506410 8765 consensus_queue.cc:260] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43199 } } peers { permanent_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43325 } } peers { permanent_uuid: "1539d7e140c545c882d1c70028fbc9f5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36777 } }
I20260923 18:13:20.507751 8765 ts_tablet_manager.cc:1428] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260923 18:13:20.508289 8732 heartbeater.cc:503] Master 127.27.75.254:38253 was elected leader, sending a full tablet report...
W20260923 18:13:20.566339 8465 tablet.cc:2369] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:13:20.613829 8600 tablet.cc:2369] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:13:20.622166 8733 tablet.cc:2369] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 18:13:20 dist-test-slave-wsx5 krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187199, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for HTTP/127.27.75.212@KRBTEST.COM
I20260923 18:13:20.805492 8772 raft_consensus.cc:482] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:13:20.805737 8772 raft_consensus.cc:504] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43199 } } peers { permanent_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43325 } } peers { permanent_uuid: "1539d7e140c545c882d1c70028fbc9f5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36777 } }
I20260923 18:13:20.807199 8772 leader_election.cc:288] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e8694cc1d424d71a3ca84af73ba3cf6 (127.27.75.195:43199), 1539d7e140c545c882d1c70028fbc9f5 (127.27.75.193:36777)
I20260923 18:13:20.815055 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.809428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:46335 (local address 127.27.75.195:43199)
0923 18:13:20.809594 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:20.809600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:20.810409 (+ 809us) server_negotiation.cc:407] Connection header received
0923 18:13:20.810600 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:20.810604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:20.810654 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:20.810698 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:20.811125 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.811910 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.813638 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.814282 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.814390 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.814445 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:20.814800 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 18:13:20.814899 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:13:20.815212 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.809588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:46591 (local address 127.27.75.193:36777)
0923 18:13:20.809710 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:20.809714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:20.810548 (+ 834us) server_negotiation.cc:407] Connection header received
0923 18:13:20.810751 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:20.810755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:20.810804 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:20.810853 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:20.811364 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.812094 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.813826 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.814427 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.814532 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.814585 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:20.814974 (+ 389us) server_negotiation.cc:299] Negotiation successful
0923 18:13:20.815070 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 18:13:20.815642 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.808307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36777 (local address 127.27.75.194:46591)
0923 18:13:20.810457 (+ 2150us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:20.810473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:20.810627 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:20.811064 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:20.811067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:20.811087 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:20.811221 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:20.811227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.812248 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.812251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:20.813638 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:20.813651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.814707 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.814711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:20.814815 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.814839 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:13:20.815380 (+ 541us) client_negotiation.cc:240] Negotiation successful
0923 18:13:20.815472 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2022,"thread_start_us":81,"threads_started":1}
I20260923 18:13:20.816442 8405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4071194a3fea42b3b3906d52e8be8096" candidate_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1539d7e140c545c882d1c70028fbc9f5" is_pre_election: true
I20260923 18:13:20.816812 8405 raft_consensus.cc:2454] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba8d54921da4fb6aa243d21a1f2bc6f in term 0.
I20260923 18:13:20.817672 8487 leader_election.cc:302] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1539d7e140c545c882d1c70028fbc9f5, 8ba8d54921da4fb6aa243d21a1f2bc6f; no voters:
I20260923 18:13:20.818050 8772 raft_consensus.cc:2781] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:13:20.818136 8772 raft_consensus.cc:482] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:13:20.818174 8772 raft_consensus.cc:3037] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:20.819530 8772 raft_consensus.cc:504] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43199 } } peers { permanent_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43325 } } peers { permanent_uuid: "1539d7e140c545c882d1c70028fbc9f5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36777 } }
I20260923 18:13:20.819985 8772 leader_election.cc:288] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [CANDIDATE]: Term 1 election: Requested vote from peers 0e8694cc1d424d71a3ca84af73ba3cf6 (127.27.75.195:43199), 1539d7e140c545c882d1c70028fbc9f5 (127.27.75.193:36777)
I20260923 18:13:20.820979 8405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4071194a3fea42b3b3906d52e8be8096" candidate_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1539d7e140c545c882d1c70028fbc9f5"
I20260923 18:13:20.821164 8405 raft_consensus.cc:3037] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:20.822696 8405 raft_consensus.cc:2454] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba8d54921da4fb6aa243d21a1f2bc6f in term 1.
I20260923 18:13:20.823441 8487 leader_election.cc:302] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1539d7e140c545c882d1c70028fbc9f5, 8ba8d54921da4fb6aa243d21a1f2bc6f; no voters:
I20260923 18:13:20.823781 8772 raft_consensus.cc:2781] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:13:20.824013 8772 raft_consensus.cc:686] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 1 LEADER]: Becoming Leader. State: Replica: 8ba8d54921da4fb6aa243d21a1f2bc6f, State: Running, Role: LEADER
I20260923 18:13:20.824430 8772 consensus_queue.cc:237] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43199 } } peers { permanent_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43325 } } peers { permanent_uuid: "1539d7e140c545c882d1c70028fbc9f5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36777 } }
I20260923 18:13:20.825565 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.807886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:43199 (local address 127.27.75.194:46335)
0923 18:13:20.810320 (+ 2434us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:20.810336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:20.810487 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:20.810832 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:20.810835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:20.810850 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:20.811002 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:20.811009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.812068 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.812072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:20.813470 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:20.813484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.814547 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.814550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:20.814652 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.814674 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:13:20.815680 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0923 18:13:20.825387 (+ 9707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2276,"thread_start_us":123,"threads_started":1}
I20260923 18:13:20.826519 8686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4071194a3fea42b3b3906d52e8be8096" candidate_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" is_pre_election: true
I20260923 18:13:20.826886 8686 raft_consensus.cc:2454] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba8d54921da4fb6aa243d21a1f2bc6f in term 0.
I20260923 18:13:20.827216 8685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4071194a3fea42b3b3906d52e8be8096" candidate_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6"
I20260923 18:13:20.827314 8685 raft_consensus.cc:3037] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:20.828673 8685 raft_consensus.cc:2454] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba8d54921da4fb6aa243d21a1f2bc6f in term 1.
I20260923 18:13:20.831791 8262 catalog_manager.cc:5495] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba8d54921da4fb6aa243d21a1f2bc6f (127.27.75.194). New cstate: current_term: 1 leader_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba8d54921da4fb6aa243d21a1f2bc6f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1539d7e140c545c882d1c70028fbc9f5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36777 } health_report { overall_health: UNKNOWN } } }
I20260923 18:13:20.909368 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.904477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.27.75.194:43325)
0923 18:13:20.904592 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:20.904597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:20.904652 (+ 55us) server_negotiation.cc:407] Connection header received
0923 18:13:20.904859 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:20.904862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:20.904936 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:20.905040 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:13:20.905467 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.906131 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.907294 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:20.907657 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:20.907764 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:20.908414 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:13:20.908874 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:13:20.908987 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:20.909094 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 18:13:20.909196 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:13:20.916164 8405 raft_consensus.cc:1261] T 4071194a3fea42b3b3906d52e8be8096 P 1539d7e140c545c882d1c70028fbc9f5 [term 1 FOLLOWER]: Refusing update from remote peer 8ba8d54921da4fb6aa243d21a1f2bc6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:13:20.916273 8685 raft_consensus.cc:1261] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6 [term 1 FOLLOWER]: Refusing update from remote peer 8ba8d54921da4fb6aa243d21a1f2bc6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:13:20.916944 8772 consensus_queue.cc:1035] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e8694cc1d424d71a3ca84af73ba3cf6" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 43199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:13:20.917173 8778 consensus_queue.cc:1035] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1539d7e140c545c882d1c70028fbc9f5" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:13:20.931124 8781 mvcc.cc:205] Tried to move back new op lower bound from 7332606774945128448 to 7332606774581571584. Current Snapshot: MvccSnapshot[applied={T|T < 7332606774945128448}]
I20260923 18:13:20.932796 8782 mvcc.cc:205] Tried to move back new op lower bound from 7332606774945128448 to 7332606774581571584. Current Snapshot: MvccSnapshot[applied={T|T < 7332606774945128448}]
I20260923 18:13:20.935909 8783 mvcc.cc:205] Tried to move back new op lower bound from 7332606774945128448 to 7332606774581571584. Current Snapshot: MvccSnapshot[applied={T|T < 7332606774945128448}]
I20260923 18:13:20.942067 8266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:13:20.942301 8266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:13:20.944965 8266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c09135e1f2249b3a7445a8406abde8b: Sending DeleteTablet for 3 replicas of tablet 4071194a3fea42b3b3906d52e8be8096
I20260923 18:13:20.945982 8666 tablet_service.cc:1514] Processing DeleteTablet for tablet 4071194a3fea42b3b3906d52e8be8096 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:20 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:51670
I20260923 18:13:20.945984 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet 4071194a3fea42b3b3906d52e8be8096 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:20 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:54042
I20260923 18:13:20.947090 8533 tablet_service.cc:1514] Processing DeleteTablet for tablet 4071194a3fea42b3b3906d52e8be8096 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:20 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:42904
I20260923 18:13:20.947515 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 8304
I20260923 18:13:20.947736 8790 tablet_replica.cc:330] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: stopping tablet replica
I20260923 18:13:20.948093 8790 raft_consensus.cc:2229] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:13:20.948477 8790 raft_consensus.cc:2258] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:13:20.948872 8792 tablet_replica.cc:330] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: stopping tablet replica
I20260923 18:13:20.949206 8792 raft_consensus.cc:2229] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:13:20.949689 8792 raft_consensus.cc:2258] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:13:20.951326 8792 ts_tablet_manager.cc:1905] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:13:20.954512 8792 ts_tablet_manager.cc:1918] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:13:20.954695 8792 log.cc:1196] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/4071194a3fea42b3b3906d52e8be8096
I20260923 18:13:20.955098 8792 ts_tablet_manager.cc:1939] T 4071194a3fea42b3b3906d52e8be8096 P 8ba8d54921da4fb6aa243d21a1f2bc6f: Deleting consensus metadata
I20260923 18:13:20.956817 8253 catalog_manager.cc:4841] TS 8ba8d54921da4fb6aa243d21a1f2bc6f (127.27.75.194:43325): tablet 4071194a3fea42b3b3906d52e8be8096 (table test-table [id=780f663fa66e43288b45735ea3c6d190]) successfully deleted
W20260923 18:13:20.957965 8252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:36777 (error 108)
I20260923 18:13:20.958783 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 8467
I20260923 18:13:20.959025 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:20.958672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36777 (local address 127.0.0.1:54046)
0923 18:13:20.958865 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:20.958935 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260923 18:13:20.959185 8252 catalog_manager.cc:4584] TS 1539d7e140c545c882d1c70028fbc9f5 (127.27.75.193:36777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4071194a3fea42b3b3906d52e8be8096: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36777: connect: Connection refused (error 111)
I20260923 18:13:20.961023 8790 ts_tablet_manager.cc:1905] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:13:20.964329 8790 ts_tablet_manager.cc:1918] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:13:20.964470 8790 log.cc:1196] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/4071194a3fea42b3b3906d52e8be8096
I20260923 18:13:20.964826 8790 ts_tablet_manager.cc:1939] T 4071194a3fea42b3b3906d52e8be8096 P 0e8694cc1d424d71a3ca84af73ba3cf6: Deleting consensus metadata
I20260923 18:13:20.966236 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 8601
W20260923 18:13:20.973620 8761 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:13:20.973233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:43199 (local address 127.0.0.1:51678)
0923 18:13:20.973430 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:20.973488 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.195:43199: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":117}
I20260923 18:13:20.973687 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 8233
I20260923 18:13:20.980289 27951 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 18:13:21.316438 27951 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 18:13:21.316593 27951 mini_ranger.cc:66] Stopping Ranger...
I20260923 18:13:21.652563 27951 mini_ranger.cc:68] Stopped Ranger
2026-09-23 18:13:21.652 UTC [7299] LOG: received smart shutdown request
2026-09-23 18:13:21.657 UTC [7299] LOG: background worker "logical replication launcher" (PID 7308) exited with exit code 1
2026-09-23 18:13:21.657 UTC [7303] LOG: shutting down
2026-09-23 18:13:21.890 UTC [7299] LOG: database system is shut down
2026-09-23T18:13:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51418 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:13:22 dist-test-slave-wsx5 krb5kdc[8802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:13:22 dist-test-slave-wsx5 krb5kdc[8802](info): set up 2 sockets
Sep 23 18:13:22 dist-test-slave-wsx5 krb5kdc[8802](info): commencing operation
krb5kdc: starting...
W20260923 18:13:22.963282 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:13:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:13:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:13:23.096594 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:43021
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:40955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:43021,127.27.75.253:38345,127.27.75.252:35685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:23.211498 8815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:23.211817 8815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:23.211874 8815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:23.217082 8815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:13:23.217236 8815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:23.217271 8815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:23.217296 8815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:13:23.217316 8815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:13:23.223459 8815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:40955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:43021,127.27.75.253:38345,127.27.75.252:35685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:23.224664 8815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:23.225839 8815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:23.236020 8823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:23.236243 8815 server_base.cc:1034] running on GCE node
W20260923 18:13:23.236181 8821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:23.236143 8820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:23.236990 8815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:23.238297 8815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:23.239507 8815 hybrid_clock.cc:648] HybridClock initialized: now 1790187203239490 us; error 53 us; skew 500 ppm
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:23.242722 8815 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:13:23.243718 8815 webserver.cc:460] Webserver started at http://127.27.75.254:36993/ using document root <none> and password file <none>
I20260923 18:13:23.244382 8815 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:23.244451 8815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:23.244665 8815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:23.246898 8815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "a22b1040c67b4dfabb92f3f58c538fc1"
format_stamp: "Formatted at 2026-09-23 18:13:23 on dist-test-slave-wsx5"
server_key: "5f66ae53527b2b12cf8b4417ef02d12a"
server_key_iv: "b39c0f9ff48d0b4a1ca803fc50bda32c"
server_key_version: "encryptionkey@0"
I20260923 18:13:23.247478 8815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "a22b1040c67b4dfabb92f3f58c538fc1"
format_stamp: "Formatted at 2026-09-23 18:13:23 on dist-test-slave-wsx5"
server_key: "5f66ae53527b2b12cf8b4417ef02d12a"
server_key_iv: "b39c0f9ff48d0b4a1ca803fc50bda32c"
server_key_version: "encryptionkey@0"
I20260923 18:13:23.251560 8815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260923 18:13:23.254228 8830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:23.255549 8815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:13:23.255676 8815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "a22b1040c67b4dfabb92f3f58c538fc1"
format_stamp: "Formatted at 2026-09-23 18:13:23 on dist-test-slave-wsx5"
server_key: "5f66ae53527b2b12cf8b4417ef02d12a"
server_key_iv: "b39c0f9ff48d0b4a1ca803fc50bda32c"
server_key_version: "encryptionkey@0"
I20260923 18:13:23.255787 8815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:23.292804 8815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:23.296278 8815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:23.296481 8815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:23.304911 8815 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:43021
I20260923 18:13:23.304975 8882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:43021 every 8 connection(s)
I20260923 18:13:23.305995 8815 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:13:23.309233 8883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:23.313452 8883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:13:23.314163 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 8815
I20260923 18:13:23.314272 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:13:23.314563 27951 external_mini_cluster.cc:1428] Setting key 754c847978510138e5a16e3dc528fb00
WARNING: no policy specified for kudu/127.27.75.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.253@KRBTEST.COM" created.
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:23.331698 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.316823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.27.75.254:43021)
0923 18:13:23.317205 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.317215 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.317264 (+ 49us) server_negotiation.cc:407] Connection header received
0923 18:13:23.317829 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.317836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.318127 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.318346 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.319289 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.320778 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.322044 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.322628 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.322764 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.326495 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.326501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.326514 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.326539 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.328175 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.328694 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.328699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.328700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.328814 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.329167 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.329171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.329172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.329603 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.329729 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.330114 (+ 385us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.330357 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
I20260923 18:13:23.331698 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.315928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43021 (local address 127.0.0.1:53012)
0923 18:13:23.316446 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.316502 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:23.317478 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:23.318474 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:23.318479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:23.318845 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:23.319114 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.319128 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.320954 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.320957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.321892 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.321903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.322808 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.322811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.322975 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.323610 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:23.323627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:23.326316 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:23.328304 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.328314 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.328326 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.328568 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.328907 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.328910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.328912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.329030 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.329719 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:23.329726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:23.329862 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 18:13:23.330012 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 18:13:23.330548 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":146,"thread_start_us":134,"threads_started":1}
I20260923 18:13:23.334522 8883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:13:23.335394 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.335105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:38345 (local address 127.0.0.1:32798)
0923 18:13:23.335237 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.335312 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 18:13:23.335677 8833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.75.253:38345: connect: Connection refused (error 111)
I20260923 18:13:23.336678 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.336391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:38345 (local address 127.0.0.1:32806)
0923 18:13:23.336562 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.336613 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260923 18:13:23.336911 8883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.253:38345: Network error: Client connection negotiation failed: client connection to 127.27.75.253:38345: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab.
Entry for principal kudu/127.27.75.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab.
I20260923 18:13:23.341753 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.253:38345
--webserver_interface=127.27.75.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab
--principal=kudu/127.27.75.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:40955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:43021,127.27.75.253:38345,127.27.75.252:35685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:13:23.386161 8883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } attempt: 1
I20260923 18:13:23.387087 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.386759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:38345 (local address 127.0.0.1:32812)
0923 18:13:23.386934 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.387002 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 18:13:23.387894 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.387670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:38345 (local address 127.0.0.1:32814)
0923 18:13:23.387785 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.387831 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 18:13:23.388019 8883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.253:38345: Network error: Client connection negotiation failed: client connection to 127.27.75.253:38345: connect: Connection refused (error 111)
W20260923 18:13:23.455468 8890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:23.455778 8890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:23.455833 8890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 18:13:23.456204 8883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } attempt: 2
I20260923 18:13:23.457166 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.456747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:38345 (local address 127.0.0.1:32828)
0923 18:13:23.456929 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.457045 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 18:13:23.458029 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.457747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:38345 (local address 127.0.0.1:32838)
0923 18:13:23.457870 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.457920 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 18:13:23.458177 8883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.253:38345: Network error: Client connection negotiation failed: client connection to 127.27.75.253:38345: connect: Connection refused (error 111)
W20260923 18:13:23.460457 8890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:13:23.460525 8890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:23.460553 8890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:23.460585 8890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:13:23.460615 8890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:13:23.466420 8890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:40955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:43021,127.27.75.253:38345,127.27.75.252:35685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab
--principal=kudu/127.27.75.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.253:38345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.75.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:23.467625 8890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:23.468695 8890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:23.478404 8896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:23.478410 8895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:23.478474 8898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:23.478978 8890 server_base.cc:1034] running on GCE node
I20260923 18:13:23.479590 8890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:23.480819 8890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:23.482029 8890 hybrid_clock.cc:648] HybridClock initialized: now 1790187203482002 us; error 49 us; skew 500 ppm
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:23.485515 8890 init.cc:377] Logged in from keytab as kudu/127.27.75.253@KRBTEST.COM (short username kudu)
I20260923 18:13:23.486552 8890 webserver.cc:460] Webserver started at http://127.27.75.253:39341/ using document root <none> and password file <none>
I20260923 18:13:23.487108 8890 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:23.487185 8890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:23.487365 8890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:23.489686 8890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/data/instance:
uuid: "65780639396a49e4ab4d910c0acc02e6"
format_stamp: "Formatted at 2026-09-23 18:13:23 on dist-test-slave-wsx5"
server_key: "2a6f230b7746ce1078b0eb8cc55fbe95"
server_key_iv: "adb7e90ee76174e8a7d687d8a150e160"
server_key_version: "encryptionkey@0"
I20260923 18:13:23.490272 8890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/wal/instance:
uuid: "65780639396a49e4ab4d910c0acc02e6"
format_stamp: "Formatted at 2026-09-23 18:13:23 on dist-test-slave-wsx5"
server_key: "2a6f230b7746ce1078b0eb8cc55fbe95"
server_key_iv: "adb7e90ee76174e8a7d687d8a150e160"
server_key_version: "encryptionkey@0"
I20260923 18:13:23.494552 8890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 18:13:23.497385 8905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:23.498980 8890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:13:23.499114 8890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/wal
uuid: "65780639396a49e4ab4d910c0acc02e6"
format_stamp: "Formatted at 2026-09-23 18:13:23 on dist-test-slave-wsx5"
server_key: "2a6f230b7746ce1078b0eb8cc55fbe95"
server_key_iv: "adb7e90ee76174e8a7d687d8a150e160"
server_key_version: "encryptionkey@0"
I20260923 18:13:23.499254 8890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:23.517675 8890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:23.521045 8890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:23.521332 8890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:23.528568 8890 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.253:38345
I20260923 18:13:23.528656 8957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.253:38345 every 8 connection(s)
I20260923 18:13:23.529644 8890 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
I20260923 18:13:23.532675 8958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:23.536638 8958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:13:23.539074 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 8890
I20260923 18:13:23.539206 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-1/wal/instance
I20260923 18:13:23.539530 27951 external_mini_cluster.cc:1428] Setting key 004509215d6ce43a529ac1a6ef7594bf
WARNING: no policy specified for kudu/127.27.75.252@KRBTEST.COM; defaulting to no policy
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:23.549336 8883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.252@KRBTEST.COM" created.
I20260923 18:13:23.551931 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.538688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.27.75.254:43021)
0923 18:13:23.538852 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.538856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.539952 (+ 1096us) server_negotiation.cc:407] Connection header received
0923 18:13:23.540994 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.540998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.541063 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.541189 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.542699 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.543532 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.544845 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.545259 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.545380 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.547948 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.547953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.547956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.547985 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.549527 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.550138 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.550141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.550143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.550238 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.550662 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.550666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.550668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.550916 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.551004 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.551640 (+ 636us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.551763 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 18:13:23.553010 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.539168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43021 (local address 127.0.0.1:53022)
0923 18:13:23.539780 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.539828 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:23.540714 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:23.541398 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:23.541408 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:23.541906 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:23.542514 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.542531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.543721 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.543725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.544709 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.544720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.545452 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.545455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.545629 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.546260 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:23.546277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:23.547706 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:23.549652 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.549660 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.549678 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.550024 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.550354 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.550358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.550360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.550546 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.551034 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:23.551041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:23.551374 (+ 333us) client_negotiation.cc:769] Sending connection context
0923 18:13:23.551557 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 18:13:23.551907 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":190,"threads_started":1}
I20260923 18:13:23.555069 8958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
I20260923 18:13:23.560806 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.549782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:38345 (local address 127.0.0.1:32840)
0923 18:13:23.549981 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.549995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:23.550080 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:23.550873 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:23.550876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:23.551068 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:23.551190 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.551197 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.552573 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.552577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.553411 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.553423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.554296 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.554301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.554420 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.554975 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:23.554991 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:23.556684 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:23.558947 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.558951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.558953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.559201 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.559542 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.559545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.559547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.559640 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.560174 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:23.560177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:23.560276 (+ 99us) client_negotiation.cc:769] Sending connection context
0923 18:13:23.560442 (+ 166us) client_negotiation.cc:240] Negotiation successful
0923 18:13:23.560617 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 18:13:23.561538 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.549927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.27.75.253:38345)
0923 18:13:23.550412 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.550424 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.550449 (+ 25us) server_negotiation.cc:407] Connection header received
0923 18:13:23.550500 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.550507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.550738 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.550864 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.551318 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.552444 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.553550 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.554114 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.554235 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.556856 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.556866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.556903 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.556926 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.558789 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.559337 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.559344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.559345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.559426 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.559728 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.559732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.559733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.560034 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.560172 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.561115 (+ 943us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.561344 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"spinlock_wait_cycles":2432,"thread_start_us":159,"threads_started":1}
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
I20260923 18:13:23.563663 8883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:13:23.564599 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.564307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60546)
0923 18:13:23.564466 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.564523 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 18:13:23.565586 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.565257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60558)
0923 18:13:23.565446 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.565507 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260923 18:13:23.565757 8883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:35685: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
I20260923 18:13:23.567036 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.555597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:38345 (local address 127.0.0.1:32848)
0923 18:13:23.555772 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.555786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:23.555865 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:23.556935 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:23.556938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:23.557435 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:23.557569 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.557578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.558885 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.558889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.559776 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.559787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.560366 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.560370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.560481 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.561102 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:23.561139 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:23.563064 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:23.565409 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.565413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.565415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.565599 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.565883 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.565886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.565888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.566117 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.566539 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:23.566542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:23.566633 (+ 91us) client_negotiation.cc:769] Sending connection context
0923 18:13:23.566718 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 18:13:23.566871 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260923 18:13:23.567142 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.555632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.27.75.253:38345)
0923 18:13:23.556594 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.556604 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.556618 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:13:23.556657 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.556660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.556709 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.556801 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.557931 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.558740 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.559886 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.560223 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.560318 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.563218 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.563222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.563224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.563247 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.565272 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.565689 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.565692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.565694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.565770 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.566227 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.566230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.566232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.566439 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.566527 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.566787 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.566953 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":888,"thread_start_us":142,"threads_started":1}
I20260923 18:13:23.567828 8958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:13:23.568572 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.568310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60574)
0923 18:13:23.568431 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.568514 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 18:13:23.568848 8909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab.
Entry for principal kudu/127.27.75.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab.
I20260923 18:13:23.569885 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.569627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60582)
0923 18:13:23.569748 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.569806 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 18:13:23.570070 8958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:35685: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
I20260923 18:13:23.570195 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.252:35685
--webserver_interface=127.27.75.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab
--principal=kudu/127.27.75.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:40955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:43021,127.27.75.253:38345,127.27.75.252:35685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:13:23.596961 8883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } attempt: 1
I20260923 18:13:23.597896 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.597543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60592)
0923 18:13:23.597724 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.597798 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 18:13:23.598650 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.598423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60602)
0923 18:13:23.598529 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.598583 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 18:13:23.598834 8883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:35685: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
I20260923 18:13:23.619369 8958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } attempt: 1
I20260923 18:13:23.620227 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.619931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60612)
0923 18:13:23.620082 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.620144 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 18:13:23.621047 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.620818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60618)
0923 18:13:23.620937 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.620985 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 18:13:23.621223 8958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:35685: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
I20260923 18:13:23.674021 8883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } attempt: 2
I20260923 18:13:23.674876 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.674531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60630)
0923 18:13:23.674707 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.674782 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 18:13:23.675612 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.675389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60642)
0923 18:13:23.675508 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.675554 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 18:13:23.675760 8883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:35685: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
W20260923 18:13:23.684666 8966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:23.685302 8966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:23.685384 8966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 18:13:23.689373 8958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } attempt: 2
W20260923 18:13:23.689774 8966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:13:23.689844 8966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:23.689869 8966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:23.689898 8966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:13:23.689924 8966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:13:23.690295 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.689929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60654)
0923 18:13:23.690147 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.690208 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260923 18:13:23.691078 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.690811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60658)
0923 18:13:23.690937 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.691009 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 18:13:23.691272 8958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:35685: Network error: Client connection negotiation failed: client connection to 127.27.75.252:35685: connect: Connection refused (error 111)
I20260923 18:13:23.695990 8966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:40955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:43021,127.27.75.253:38345,127.27.75.252:35685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab
--principal=kudu/127.27.75.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.252:35685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.75.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:23.697067 8966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:23.698175 8966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:23.706874 8972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:23.706874 8971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:23.707296 8966 server_base.cc:1034] running on GCE node
W20260923 18:13:23.706929 8974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:23.707938 8966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:23.709223 8966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:23.710399 8966 hybrid_clock.cc:648] HybridClock initialized: now 1790187203710369 us; error 50 us; skew 500 ppm
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:23.713459 8966 init.cc:377] Logged in from keytab as kudu/127.27.75.252@KRBTEST.COM (short username kudu)
I20260923 18:13:23.714413 8966 webserver.cc:460] Webserver started at http://127.27.75.252:33671/ using document root <none> and password file <none>
I20260923 18:13:23.714955 8966 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:23.715034 8966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:23.715247 8966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:23.717362 8966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/data/instance:
uuid: "a882f986d4ee41afa8c6bedfe520d96f"
format_stamp: "Formatted at 2026-09-23 18:13:23 on dist-test-slave-wsx5"
server_key: "848d8698b92f3610010a7f3c2a9396d7"
server_key_iv: "e56e7b197c48cc695c0d7ce479bb5d30"
server_key_version: "encryptionkey@0"
I20260923 18:13:23.717931 8966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/wal/instance:
uuid: "a882f986d4ee41afa8c6bedfe520d96f"
format_stamp: "Formatted at 2026-09-23 18:13:23 on dist-test-slave-wsx5"
server_key: "848d8698b92f3610010a7f3c2a9396d7"
server_key_iv: "e56e7b197c48cc695c0d7ce479bb5d30"
server_key_version: "encryptionkey@0"
I20260923 18:13:23.721748 8966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:13:23.724282 8981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:23.725719 8966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:13:23.725858 8966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/wal
uuid: "a882f986d4ee41afa8c6bedfe520d96f"
format_stamp: "Formatted at 2026-09-23 18:13:23 on dist-test-slave-wsx5"
server_key: "848d8698b92f3610010a7f3c2a9396d7"
server_key_iv: "e56e7b197c48cc695c0d7ce479bb5d30"
server_key_version: "encryptionkey@0"
I20260923 18:13:23.726001 8966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:23.750244 8966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:23.753554 8966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:23.753764 8966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:23.760637 8966 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.252:35685
I20260923 18:13:23.760665 9033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.252:35685 every 8 connection(s)
I20260923 18:13:23.761683 8966 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
I20260923 18:13:23.764536 9034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:23.767107 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 8966
I20260923 18:13:23.767235 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790187042487232-27951-0/minicluster-data/master-2/wal/instance
I20260923 18:13:23.767521 27951 external_mini_cluster.cc:1428] Setting key aea7acb293051c3a2b20551600b9bcfd
I20260923 18:13:23.769012 9034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:13:23.774945 8883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } attempt: 3
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
I20260923 18:13:23.782084 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.770889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.27.75.254:43021)
0923 18:13:23.771019 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.771024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.771388 (+ 364us) server_negotiation.cc:407] Connection header received
0923 18:13:23.771572 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.771576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.771626 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.771713 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.772833 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.773620 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.774952 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.775285 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.775376 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.778416 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.778420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.778423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.778447 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.779946 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.780472 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.780476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.780478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.780573 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.780931 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.780934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.780936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.781184 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.781280 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.781755 (+ 475us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.781887 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 18:13:23.782413 8958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } attempt: 3
I20260923 18:13:23.783680 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.769083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.27.75.252:35685)
0923 18:13:23.769589 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.769607 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.769645 (+ 38us) server_negotiation.cc:407] Connection header received
0923 18:13:23.770306 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.770324 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.770677 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.771008 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.771837 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.773545 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.774526 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.775037 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.775635 (+ 598us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.777899 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.777906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.777920 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.777945 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.780101 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.780513 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.780519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.780521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.780656 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.780955 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.780958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.780959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.781513 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.781699 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.781934 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.782501 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":346,"spinlock_wait_cycles":3840,"thread_start_us":128,"threads_started":1}
I20260923 18:13:23.783692 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.770761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:43021 (local address 127.0.0.1:53036)
0923 18:13:23.771265 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.771313 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:23.771472 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:23.771819 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:23.771825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:23.772350 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:23.772647 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.772664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.773761 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.773765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.774804 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.774814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.775541 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.775546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.775669 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.776211 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:23.776231 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:23.778247 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:23.780053 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.780062 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.780074 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.780357 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.780677 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.780680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.780682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.780818 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.781296 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:23.781303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:23.781483 (+ 180us) client_negotiation.cc:769] Sending connection context
0923 18:13:23.782283 (+ 800us) client_negotiation.cc:240] Negotiation successful
0923 18:13:23.782729 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":1,"spinlock_wait_cycles":53504,"thread_start_us":143,"threads_started":1}
I20260923 18:13:23.784984 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.775377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60680)
0923 18:13:23.775886 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.775903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:23.776202 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:23.776505 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:23.776508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:23.776730 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:23.776860 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.776869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.777742 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.777746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.778490 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.778501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.779029 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.779032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.779123 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.779527 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:23.779548 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:23.780908 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:23.783344 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.783347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.783350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.783586 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.783916 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.783919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.783921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.784007 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.784419 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:23.784421 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:23.784515 (+ 94us) client_negotiation.cc:769] Sending connection context
0923 18:13:23.784601 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 18:13:23.784704 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260923 18:13:23.785451 9034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:13:23.785584 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.775451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.27.75.252:35685)
0923 18:13:23.775777 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.775789 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.775976 (+ 187us) server_negotiation.cc:407] Connection header received
0923 18:13:23.776315 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.776318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.776368 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.776468 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.776958 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.777616 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.778606 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.778932 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.779028 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.781024 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.781027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.781030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.781052 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.783222 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.783707 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.783710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.783711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.783812 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.784115 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.784118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.784119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.784314 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.785196 (+ 882us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.785272 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.785372 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"spinlock_wait_cycles":11648,"thread_start_us":110,"threads_started":1}
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
I20260923 18:13:23.789880 8883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1: Bootstrap starting.
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
I20260923 18:13:23.793035 8883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:23.794042 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.782831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60686)
0923 18:13:23.783018 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.783032 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:23.783122 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:23.785108 (+ 1986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:23.785112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:23.785320 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:23.785432 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.785440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.786364 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.786368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.787043 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.787055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.788010 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.788013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.788094 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.788600 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:23.788614 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:23.790129 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:23.792015 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.792018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.792021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.792261 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.792560 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.792563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.792566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.792662 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.793184 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:23.793187 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:23.793303 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 18:13:23.793415 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 18:13:23.793786 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260923 18:13:23.794580 8883 log.cc:824] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:23.795109 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.782880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.27.75.252:35685)
0923 18:13:23.784773 (+ 1893us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.784777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.784789 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:13:23.784819 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.784821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.784862 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.784941 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.785563 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.786233 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.787501 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.787794 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.787882 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.790285 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.790289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.790292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.790317 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.791882 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.792369 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.792373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.792375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.792457 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.792759 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.792762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.792763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.793021 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.793112 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.794766 (+ 1654us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.794926 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1837,"thread_start_us":107,"threads_started":1}
I20260923 18:13:23.796530 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.786317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.27.75.253:38345)
0923 18:13:23.786588 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.786592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.786603 (+ 11us) server_negotiation.cc:407] Connection header received
0923 18:13:23.786634 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.786637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.786677 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.786738 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.787167 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.787811 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.788842 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.789381 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.789452 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.792106 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.792110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.792113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.792141 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.794527 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.795117 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.795121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.795123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.795214 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.795518 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.795522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.795523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.795813 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.795888 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.796265 (+ 377us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.796395 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260923 18:13:23.796943 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.786001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:38345 (local address 127.0.0.1:32852)
0923 18:13:23.786426 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.786441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:23.786509 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:23.786778 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:23.786781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:23.786953 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:23.787044 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.787051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.787942 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.787945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.788713 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.788724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.789608 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.789612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.789717 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.790309 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:23.790323 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:23.791930 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:23.794672 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.794675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.794678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.795010 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.795322 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.795325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.795327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.795412 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.795960 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:23.795963 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:23.796065 (+ 102us) client_negotiation.cc:769] Sending connection context
0923 18:13:23.796660 (+ 595us) client_negotiation.cc:240] Negotiation successful
0923 18:13:23.796805 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362}
I20260923 18:13:23.797727 8883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1: No bootstrap required, opened a new log
I20260923 18:13:23.798709 8958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6: Bootstrap starting.
I20260923 18:13:23.799857 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.786369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.27.75.254:43021)
0923 18:13:23.787237 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.787242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.787254 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:13:23.787286 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.787288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.787330 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.787405 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.788149 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.789324 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.790959 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.791531 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.791624 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.795690 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.795695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.795697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.795728 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.797923 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.798486 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.798490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.798492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.798590 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.798933 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.798936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.798938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.799178 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.799358 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.799575 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.799703 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796}
I20260923 18:13:23.801688 8958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:23.801867 8883 raft_consensus.cc:348] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } }
I20260923 18:13:23.802127 8883 raft_consensus.cc:374] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:23.802181 8883 raft_consensus.cc:729] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22b1040c67b4dfabb92f3f58c538fc1, State: Initialized, Role: FOLLOWER
I20260923 18:13:23.802635 8958 log.cc:824] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:23.802762 8883 consensus_queue.cc:260] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } }
I20260923 18:13:23.802951 9034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:13:23.803577 9042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } } }
I20260923 18:13:23.803787 9042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:13:23.804881 8883 sys_catalog.cc:564] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
I20260923 18:13:23.808506 8958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6: No bootstrap required, opened a new log
Sep 23 18:13:23 dist-test-slave-wsx5 krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187203, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
I20260923 18:13:23.812754 8958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } }
I20260923 18:13:23.813032 8958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:23.813102 8958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65780639396a49e4ab4d910c0acc02e6, State: Initialized, Role: FOLLOWER
I20260923 18:13:23.813680 8958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } }
I20260923 18:13:23.815068 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.801511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.27.75.253:38345)
0923 18:13:23.801635 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.801639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.801654 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:13:23.801696 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.801699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.801745 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.801832 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.802687 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.803977 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.805105 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.805469 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.805562 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.810202 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.810206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.810209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.810235 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.812701 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.813293 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.813297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.813299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.813389 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.813688 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.813691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.813693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.813908 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.814092 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.814792 (+ 700us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.814923 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:13:23.815212 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.805013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:35685 (local address 127.0.0.1:60690)
0923 18:13:23.807297 (+ 2284us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:23.807315 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:23.807425 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:23.807678 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:23.807681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:23.807841 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:23.807912 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.807918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.808728 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.808731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.809410 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:23.809423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.809918 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.809922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:23.809991 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.810441 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:23.810455 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:23.811976 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:23.813805 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.813808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.813810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.814024 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.814261 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:23.814264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:23.814266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:23.814334 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:23.814749 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:23.814752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:23.814839 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 18:13:23.814934 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 18:13:23.815061 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2190}
I20260923 18:13:23.815565 8958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:13:23.815659 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.803733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.27.75.252:35685)
0923 18:13:23.803872 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.803876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.807500 (+ 3624us) server_negotiation.cc:407] Connection header received
0923 18:13:23.807527 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.807530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.807566 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.807621 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.808037 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.808621 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.809537 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.809805 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.809893 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.812125 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.812130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.812132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.812159 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.813695 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.814111 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.814114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.814116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.814182 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.814414 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.814416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.814418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.814618 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.814717 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.815408 (+ 691us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.815519 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:13:23.811882 8815 master_runner.cc:186] Error getting master registration for 127.27.75.253:38345: OK, instance_id { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" instance_seqno: 1790187203526512 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 18:13:23.816351 8815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:13:23.819298 9034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f: Bootstrap starting.
I20260923 18:13:23.819628 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:13:23.818595 9055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } } }
I20260923 18:13:23.821224 9055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 18:13:23.821420 9066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:13:23.821491 9066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260923 18:13:23.821627 9054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:13:23.821696 9054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:13:23.822121 9034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:23.823011 9034 log.cc:824] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:23.826913 8890 master_runner.cc:186] Error getting master registration for 127.27.75.252:35685: OK, instance_id { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" instance_seqno: 1790187203758431 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 18:13:23.830917 8890 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:13:23.837071 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.825547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.27.75.254:43021)
0923 18:13:23.825847 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.825851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.825973 (+ 122us) server_negotiation.cc:407] Connection header received
0923 18:13:23.826178 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.826181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.826232 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.826280 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.827325 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.828278 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.829586 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.829921 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.830005 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.832799 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.832804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.832807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.832841 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.835076 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.835586 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.835590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.835592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.835701 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.836012 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.836016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.836017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.836249 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.836369 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.836768 (+ 399us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.836894 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260923 18:13:23.840736 9034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f: No bootstrap required, opened a new log
I20260923 18:13:23.842274 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.826093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32862 (local address 127.27.75.253:38345)
0923 18:13:23.826204 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.826208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.831301 (+ 5093us) server_negotiation.cc:407] Connection header received
0923 18:13:23.831608 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.831612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.831680 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.831786 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.832824 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.834331 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.835552 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.835867 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.835956 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.837925 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.837929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.837931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.837963 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.840269 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.840747 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.840751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.840752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.840844 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.841289 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.841293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.841294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.841516 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.841613 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.841972 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.842102 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 18:13:23.845083 9034 raft_consensus.cc:348] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } }
I20260923 18:13:23.845588 9034 raft_consensus.cc:374] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:23.845656 9034 raft_consensus.cc:729] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a882f986d4ee41afa8c6bedfe520d96f, State: Initialized, Role: FOLLOWER
I20260923 18:13:23.846158 9034 consensus_queue.cc:260] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } }
I20260923 18:13:23.846764 9077 sys_catalog.cc:455] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } } }
I20260923 18:13:23.846973 9077 sys_catalog.cc:458] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:13:23.847616 9034 sys_catalog.cc:564] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:13:23.854816 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:23.826527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.27.75.252:35685)
0923 18:13:23.826642 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:23.826645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:23.833460 (+ 6815us) server_negotiation.cc:407] Connection header received
0923 18:13:23.833598 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:23.833602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:23.833658 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:23.833719 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:23.837274 (+ 3555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.837980 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.839177 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:23.839511 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:23.839590 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:23.850694 (+ 11104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:23.850699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:23.850702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:23.850735 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:23.852458 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.853289 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.853293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.853295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.853401 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:23.853726 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:23.853730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:23.853732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:23.853976 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:23.854097 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:23.854515 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 18:13:23.854660 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260923 18:13:23.856997 9088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260923 18:13:23.857177 8966 master_runner.cc:263] No leader master found from master a882f986d4ee41afa8c6bedfe520d96f
I20260923 18:13:23.857236 8966 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 18:13:23.857194 9088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260923 18:13:24.005276 8987 tablet.cc:2369] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:13:24.023059 8911 tablet.cc:2369] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:13:24.043737 9042 raft_consensus.cc:482] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:13:24.043964 9042 raft_consensus.cc:504] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } }
I20260923 18:13:24.045231 9042 leader_election.cc:288] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65780639396a49e4ab4d910c0acc02e6 (127.27.75.253:38345), a882f986d4ee41afa8c6bedfe520d96f (127.27.75.252:35685)
I20260923 18:13:24.046034 8932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65780639396a49e4ab4d910c0acc02e6" is_pre_election: true
I20260923 18:13:24.046068 9008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a882f986d4ee41afa8c6bedfe520d96f" is_pre_election: true
I20260923 18:13:24.046366 9008 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22b1040c67b4dfabb92f3f58c538fc1 in term 0.
I20260923 18:13:24.046362 8932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22b1040c67b4dfabb92f3f58c538fc1 in term 0.
I20260923 18:13:24.046833 8833 leader_election.cc:302] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 65780639396a49e4ab4d910c0acc02e6, a22b1040c67b4dfabb92f3f58c538fc1, a882f986d4ee41afa8c6bedfe520d96f; no voters:
I20260923 18:13:24.047128 9042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:13:24.047209 9042 raft_consensus.cc:482] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:13:24.047250 9042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [term 0 FOLLOWER]: Advancing to term 1
W20260923 18:13:24.048339 8836 tablet.cc:2369] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:13:24.048420 9042 raft_consensus.cc:504] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } }
I20260923 18:13:24.048799 9042 leader_election.cc:288] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [CANDIDATE]: Term 1 election: Requested vote from peers 65780639396a49e4ab4d910c0acc02e6 (127.27.75.253:38345), a882f986d4ee41afa8c6bedfe520d96f (127.27.75.252:35685)
I20260923 18:13:24.049189 9008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a882f986d4ee41afa8c6bedfe520d96f"
I20260923 18:13:24.049206 8932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65780639396a49e4ab4d910c0acc02e6"
I20260923 18:13:24.049324 8932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:24.049325 9008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:24.050542 8932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22b1040c67b4dfabb92f3f58c538fc1 in term 1.
I20260923 18:13:24.050542 9008 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22b1040c67b4dfabb92f3f58c538fc1 in term 1.
I20260923 18:13:24.050850 8834 leader_election.cc:302] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22b1040c67b4dfabb92f3f58c538fc1, a882f986d4ee41afa8c6bedfe520d96f; no voters:
I20260923 18:13:24.051123 9042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:13:24.051352 9042 raft_consensus.cc:686] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [term 1 LEADER]: Becoming Leader. State: Replica: a22b1040c67b4dfabb92f3f58c538fc1, State: Running, Role: LEADER
I20260923 18:13:24.051674 9042 consensus_queue.cc:237] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } }
I20260923 18:13:24.053598 9090 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a22b1040c67b4dfabb92f3f58c538fc1. Latest consensus state: current_term: 1 leader_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } } }
I20260923 18:13:24.053725 9090 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:24.054078 9092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:13:24.054958 9092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:13:24.059648 9008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [term 1 FOLLOWER]: Refusing update from remote peer a22b1040c67b4dfabb92f3f58c538fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:13:24.059643 8932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [term 1 FOLLOWER]: Refusing update from remote peer a22b1040c67b4dfabb92f3f58c538fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:13:24.060241 9042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:13:24.060426 9090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:13:24.064069 9055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a22b1040c67b4dfabb92f3f58c538fc1. Latest consensus state: current_term: 1 leader_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } } }
I20260923 18:13:24.064073 9077 sys_catalog.cc:455] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a22b1040c67b4dfabb92f3f58c538fc1. Latest consensus state: current_term: 1 leader_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } } }
I20260923 18:13:24.064193 9077 sys_catalog.cc:458] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:13:24.064191 9055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:13:24.066947 9090 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } } }
I20260923 18:13:24.067081 9090 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:24.067137 9042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } } }
I20260923 18:13:24.067168 9093 mvcc.cc:205] Tried to move back new op lower bound from 7332606787821907968 to 7332606787798188032. Current Snapshot: MvccSnapshot[applied={T|T < 7332606787821907968}]
I20260923 18:13:24.067215 9042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1 [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:24.067348 9077 sys_catalog.cc:455] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } } }
I20260923 18:13:24.067408 9095 mvcc.cc:205] Tried to move back new op lower bound from 7332606787821907968 to 7332606787798188032. Current Snapshot: MvccSnapshot[applied={T|T < 7332606787821907968}]
I20260923 18:13:24.067474 9077 sys_catalog.cc:458] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:13:24.068320 9092 catalog_manager.cc:1347] Generated new cluster ID: 4f29868f383a40808e96e7f107292c22
I20260923 18:13:24.068384 9092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:13:24.068359 9055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22b1040c67b4dfabb92f3f58c538fc1" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 43021 } } peers { permanent_uuid: "65780639396a49e4ab4d910c0acc02e6" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 38345 } } peers { permanent_uuid: "a882f986d4ee41afa8c6bedfe520d96f" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 35685 } } }
I20260923 18:13:24.068459 9055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:13:24.095805 9092 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:13:24.096951 9092 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:13:24.117790 9092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a22b1040c67b4dfabb92f3f58c538fc1: Generated new TSK 0
I20260923 18:13:24.118880 9092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:13:24.822840 9066 catalog_manager.cc:1259] Loaded cluster ID: 4f29868f383a40808e96e7f107292c22
I20260923 18:13:24.822925 9066 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6: loading cluster ID for follower catalog manager: success
I20260923 18:13:24.824620 9066 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6: acquiring CA information for follower catalog manager: success
I20260923 18:13:24.825905 9066 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65780639396a49e4ab4d910c0acc02e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:13:24.858652 9088 catalog_manager.cc:1259] Loaded cluster ID: 4f29868f383a40808e96e7f107292c22
I20260923 18:13:24.858740 9088 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f: loading cluster ID for follower catalog manager: success
I20260923 18:13:24.860639 9088 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f: acquiring CA information for follower catalog manager: success
I20260923 18:13:24.861914 9088 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a882f986d4ee41afa8c6bedfe520d96f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:13:25.195919 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 8815
I20260923 18:13:25.201824 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 8890
I20260923 18:13:25.206777 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 8966
2026-09-23T18:13:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3218 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:13:25 dist-test-slave-wsx5 krb5kdc[9108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:13:25 dist-test-slave-wsx5 krb5kdc[9108](info): set up 2 sockets
Sep 23 18:13:25 dist-test-slave-wsx5 krb5kdc[9108](info): commencing operation
krb5kdc: starting...
W20260923 18:13:26.183328 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:13:26 dist-test-slave-wsx5 krb5kdc[9108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:13:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:13:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:13:26.308660 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:36439
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:36439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:26.423851 9121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:26.424158 9121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:26.424201 9121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:26.428599 9121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:13:26.428694 9121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:26.428715 9121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:26.428733 9121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:13:26.428748 9121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:13:26.434465 9121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:36439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:36439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:26.435595 9121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:26.436655 9121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:26.445753 9127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:26.445755 9129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:26.445806 9121 server_base.cc:1034] running on GCE node
W20260923 18:13:26.445753 9126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:26.446636 9121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:26.447841 9121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:26.449033 9121 hybrid_clock.cc:648] HybridClock initialized: now 1790187206449007 us; error 42 us; skew 500 ppm
Sep 23 18:13:26 dist-test-slave-wsx5 krb5kdc[9108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187206, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:26.451922 9121 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:13:26.452919 9121 webserver.cc:460] Webserver started at http://127.27.75.254:42045/ using document root <none> and password file <none>
I20260923 18:13:26.453542 9121 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:26.453613 9121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:26.453819 9121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:26.455986 9121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "27abf9e6c86c408f8f69e7cb20e63a35"
format_stamp: "Formatted at 2026-09-23 18:13:26 on dist-test-slave-wsx5"
server_key: "198fa96e33625f83e4a621ca4b071725"
server_key_iv: "e2210970597f043e4c3a46adcc978d68"
server_key_version: "encryptionkey@0"
I20260923 18:13:26.456560 9121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "27abf9e6c86c408f8f69e7cb20e63a35"
format_stamp: "Formatted at 2026-09-23 18:13:26 on dist-test-slave-wsx5"
server_key: "198fa96e33625f83e4a621ca4b071725"
server_key_iv: "e2210970597f043e4c3a46adcc978d68"
server_key_version: "encryptionkey@0"
I20260923 18:13:26.460506 9121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:13:26.463613 9136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:26.464937 9121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:13:26.465071 9121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "27abf9e6c86c408f8f69e7cb20e63a35"
format_stamp: "Formatted at 2026-09-23 18:13:26 on dist-test-slave-wsx5"
server_key: "198fa96e33625f83e4a621ca4b071725"
server_key_iv: "e2210970597f043e4c3a46adcc978d68"
server_key_version: "encryptionkey@0"
I20260923 18:13:26.465256 9121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:26.480969 9121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:26.484282 9121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:26.484477 9121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:26.491540 9121 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:36439
I20260923 18:13:26.491612 9188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:36439 every 8 connection(s)
I20260923 18:13:26.492527 9121 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:13:26.495388 9189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:26.495410 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 9121
I20260923 18:13:26.495515 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:13:26.495810 27951 external_mini_cluster.cc:1428] Setting key 33a58344194875a9ce8c0be0612d3d0f
I20260923 18:13:26.502305 9189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35: Bootstrap starting.
Sep 23 18:13:26 dist-test-slave-wsx5 krb5kdc[9108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:26.504766 9189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:26.505586 9189 log.cc:824] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:26.507501 9189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35: No bootstrap required, opened a new log
I20260923 18:13:26.509908 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:26.497359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.27.75.254:36439)
0923 18:13:26.497842 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:26.497856 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:26.497893 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:13:26.498563 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:26.498582 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:26.498925 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:26.499250 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:26.500076 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:26.501161 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:26.502024 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:26.502559 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:26.502714 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:26.505345 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:26.505350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:26.505364 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:26.505396 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:26.507019 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:26.507454 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:26.507460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:26.507464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:26.507578 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:26.507872 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:26.507874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:26.507875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:26.508321 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:26.508465 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:26.508763 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 18:13:26.509043 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":150,"threads_started":1}
I20260923 18:13:26.511528 9189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27abf9e6c86c408f8f69e7cb20e63a35" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 36439 } }
I20260923 18:13:26.511798 9189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:26.511873 9189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27abf9e6c86c408f8f69e7cb20e63a35, State: Initialized, Role: FOLLOWER
I20260923 18:13:26.512324 9189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27abf9e6c86c408f8f69e7cb20e63a35" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 36439 } }
I20260923 18:13:26.512487 9189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:13:26.512560 9189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:13:26.512655 9189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:26.512884 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:13:26.513669 9189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27abf9e6c86c408f8f69e7cb20e63a35" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 36439 } }
I20260923 18:13:26.514011 9189 leader_election.cc:302] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27abf9e6c86c408f8f69e7cb20e63a35; no voters:
I20260923 18:13:26.514360 9189 leader_election.cc:288] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:13:26.514509 9194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:13:26.514740 9194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [term 1 LEADER]: Becoming Leader. State: Replica: 27abf9e6c86c408f8f69e7cb20e63a35, State: Running, Role: LEADER
I20260923 18:13:26.515129 9194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27abf9e6c86c408f8f69e7cb20e63a35" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 36439 } }
I20260923 18:13:26.515663 9189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:13:26.516786 9195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27abf9e6c86c408f8f69e7cb20e63a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27abf9e6c86c408f8f69e7cb20e63a35" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 36439 } } }
I20260923 18:13:26.516763 9196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27abf9e6c86c408f8f69e7cb20e63a35. Latest consensus state: current_term: 1 leader_uuid: "27abf9e6c86c408f8f69e7cb20e63a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27abf9e6c86c408f8f69e7cb20e63a35" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 36439 } } }
I20260923 18:13:26.516899 9195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:26.516899 9196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35 [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:26.517323 9203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:13:26.520489 9203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:13:26.526485 9203 catalog_manager.cc:1347] Generated new cluster ID: afae1c32477a442d80db98b08968be12
I20260923 18:13:26.526566 9203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:13:26.544102 9203 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:13:26.545166 9203 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:13:26.550298 9203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27abf9e6c86c408f8f69e7cb20e63a35: Generated new TSK 0
I20260923 18:13:26.551069 9203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:13:26.570675 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 9121
2026-09-23T18:13:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1366 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:13:26 dist-test-slave-wsx5 krb5kdc[9216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:13:26 dist-test-slave-wsx5 krb5kdc[9216](info): set up 2 sockets
Sep 23 18:13:26 dist-test-slave-wsx5 krb5kdc[9216](info): commencing operation
krb5kdc: starting...
W20260923 18:13:27.535439 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:13:27 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:13:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:13:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:13:27.662001 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39307
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:27.775712 9229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:27.776048 9229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:27.776109 9229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:27.780284 9229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:13:27.780357 9229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:27.780386 9229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:27.780406 9229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:13:27.780427 9229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:13:27.786373 9229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:39307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:27.787634 9229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:27.788708 9229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:27.797802 9234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:27.797847 9229 server_base.cc:1034] running on GCE node
W20260923 18:13:27.797803 9237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:27.797806 9235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:27.798635 9229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:27.799762 9229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:27.800942 9229 hybrid_clock.cc:648] HybridClock initialized: now 1790187207800913 us; error 41 us; skew 500 ppm
Sep 23 18:13:27 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187207, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:27.803746 9229 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:13:27.804744 9229 webserver.cc:460] Webserver started at http://127.27.75.254:35323/ using document root <none> and password file <none>
I20260923 18:13:27.805346 9229 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:27.805409 9229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:27.805608 9229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:27.807659 9229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "38f41e55c743454a9a9d02f3805702f4"
format_stamp: "Formatted at 2026-09-23 18:13:27 on dist-test-slave-wsx5"
server_key: "592d78a1e671c48c6ebc8a8f046502e1"
server_key_iv: "649b3339777494cd204735e3bf1e3860"
server_key_version: "encryptionkey@0"
I20260923 18:13:27.808197 9229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "38f41e55c743454a9a9d02f3805702f4"
format_stamp: "Formatted at 2026-09-23 18:13:27 on dist-test-slave-wsx5"
server_key: "592d78a1e671c48c6ebc8a8f046502e1"
server_key_iv: "649b3339777494cd204735e3bf1e3860"
server_key_version: "encryptionkey@0"
I20260923 18:13:27.812496 9229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 18:13:27.815775 9244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:27.817406 9229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:13:27.817533 9229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "38f41e55c743454a9a9d02f3805702f4"
format_stamp: "Formatted at 2026-09-23 18:13:27 on dist-test-slave-wsx5"
server_key: "592d78a1e671c48c6ebc8a8f046502e1"
server_key_iv: "649b3339777494cd204735e3bf1e3860"
server_key_version: "encryptionkey@0"
I20260923 18:13:27.817655 9229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:27.834473 9229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:27.837770 9229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:27.837970 9229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:27.844467 9229 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:39307
I20260923 18:13:27.844736 9296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:39307 every 8 connection(s)
I20260923 18:13:27.845486 9229 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:13:27.848263 9297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:27.848572 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 9229
I20260923 18:13:27.848683 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:13:27.848954 27951 external_mini_cluster.cc:1428] Setting key 7307528bcc5beea64496a0a52e4f28cb
I20260923 18:13:27.855790 9297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Bootstrap starting.
I20260923 18:13:27.858407 9297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Neither blocks nor log segments found. Creating new log.
Sep 23 18:13:27 dist-test-slave-wsx5 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:27.859319 9297 log.cc:824] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:27.861399 9297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: No bootstrap required, opened a new log
I20260923 18:13:27.865672 9297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:27.865928 9297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:27.866005 9297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f41e55c743454a9a9d02f3805702f4, State: Initialized, Role: FOLLOWER
I20260923 18:13:27.866125 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:27.850397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.27.75.254:39307)
0923 18:13:27.850851 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:27.850871 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:27.850915 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:13:27.851686 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:27.851712 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:27.852187 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:27.852585 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:27.853474 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:27.854965 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:27.855883 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:27.856568 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:27.856732 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:27.859443 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:27.859450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:27.859471 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:27.859510 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:27.861269 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:27.861941 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:27.861949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:27.861956 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:27.862218 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:27.862573 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:27.862577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:27.862579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:27.863424 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:27.863702 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:27.864043 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 18:13:27.864534 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260923 18:13:27.866562 9297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:27.866699 9297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:13:27.866757 9297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:13:27.866859 9297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:27.868005 9297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:27.868399 9297 leader_election.cc:302] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38f41e55c743454a9a9d02f3805702f4; no voters:
I20260923 18:13:27.868824 9297 leader_election.cc:288] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:13:27.869023 9302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:13:27.869335 9302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 1 LEADER]: Becoming Leader. State: Replica: 38f41e55c743454a9a9d02f3805702f4, State: Running, Role: LEADER
I20260923 18:13:27.869655 9302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:27.870208 9297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:13:27.871380 9304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f41e55c743454a9a9d02f3805702f4. Latest consensus state: current_term: 1 leader_uuid: "38f41e55c743454a9a9d02f3805702f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } } }
I20260923 18:13:27.871515 9304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:27.871799 9303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38f41e55c743454a9a9d02f3805702f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } } }
I20260923 18:13:27.871901 9311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:13:27.871906 9303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:27.875227 9311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:13:27.881948 9311 catalog_manager.cc:1347] Generated new cluster ID: f49217422d81469d9e4bf41aa928b8b0
I20260923 18:13:27.882045 9311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:13:27.918280 9311 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:13:27.919292 9311 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:13:27.942042 9311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Generated new TSK 0
I20260923 18:13:27.942902 9311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:13:27.985396 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39307
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:28.105003 9323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:28.105356 9323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:28.105405 9323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:28.109544 9323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:28.109620 9323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:28.109680 9323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:13:28.115152 9323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:28.116300 9323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:28.117372 9323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:28.127147 9329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:28.127153 9328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:28.127619 9323 server_base.cc:1034] running on GCE node
W20260923 18:13:28.127779 9331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:28.128173 9323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:28.128790 9323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:28.129962 9323 hybrid_clock.cc:648] HybridClock initialized: now 1790187208129922 us; error 53 us; skew 500 ppm
Sep 23 18:13:28 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:28.132649 9323 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:13:28.133538 9323 webserver.cc:460] Webserver started at http://127.27.75.193:35903/ using document root <none> and password file <none>
I20260923 18:13:28.134087 9323 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:28.134148 9323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:28.134325 9323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:28.136332 9323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "1b28b788863641d884c16c6e41fc03fb"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "b1adbf168f0cc5818fa287b94c98a2bc"
server_key_iv: "cfc170784cfc775ad5063fbabf6bc96c"
server_key_version: "encryptionkey@0"
I20260923 18:13:28.136865 9323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "1b28b788863641d884c16c6e41fc03fb"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "b1adbf168f0cc5818fa287b94c98a2bc"
server_key_iv: "cfc170784cfc775ad5063fbabf6bc96c"
server_key_version: "encryptionkey@0"
I20260923 18:13:28.140714 9323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:13:28.143298 9338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:28.144655 9323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:13:28.144809 9323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "1b28b788863641d884c16c6e41fc03fb"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "b1adbf168f0cc5818fa287b94c98a2bc"
server_key_iv: "cfc170784cfc775ad5063fbabf6bc96c"
server_key_version: "encryptionkey@0"
I20260923 18:13:28.144929 9323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:28.162588 9323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:28.166003 9323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:28.166167 9323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:28.166718 9323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:28.167536 9323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:28.167583 9323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:28.167625 9323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:28.167639 9323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:28.176147 9323 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:44461
I20260923 18:13:28.176247 9451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:44461 every 8 connection(s)
I20260923 18:13:28.177100 9323 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:13:28.182694 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 9323
I20260923 18:13:28.182826 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:13:28.183130 27951 external_mini_cluster.cc:1428] Setting key 9b87953ca526efaba588ad9366b28896
Sep 23 18:13:28 dist-test-slave-wsx5 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:13:28.192637 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.179042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:53515 (local address 127.27.75.254:39307)
0923 18:13:28.179190 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:28.179195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:28.180215 (+ 1020us) server_negotiation.cc:407] Connection header received
0923 18:13:28.181707 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:28.181710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:28.181768 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:28.181851 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:28.183609 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.184294 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.185716 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.186060 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.186172 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.188835 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:28.188838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:28.188840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:28.188870 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:28.190433 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:28.191016 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:28.191019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:28.191020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:28.191111 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:28.191457 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:28.191460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:28.191462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:28.191694 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:28.191785 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:28.192358 (+ 573us) server_negotiation.cc:299] Negotiation successful
0923 18:13:28.192506 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:13:28.193574 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.179455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:53515)
0923 18:13:28.180048 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:28.180099 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:28.181458 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:28.182187 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:28.182200 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:28.182631 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:28.183416 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.183438 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.184459 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.184463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.185571 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.185583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.186225 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.186228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.186382 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.186975 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:28.186993 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:28.188629 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:28.190583 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:28.190590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:28.190610 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:28.190906 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:28.191204 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:28.191207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:28.191210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:28.191362 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:28.191797 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:28.191805 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:28.192070 (+ 265us) client_negotiation.cc:769] Sending connection context
0923 18:13:28.192269 (+ 199us) client_negotiation.cc:240] Negotiation successful
0923 18:13:28.192569 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":131,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:13:28.194952 9452 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39307
I20260923 18:13:28.195255 9452 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:28.195765 9452 heartbeater.cc:511] Master 127.27.75.254:39307 requested a full tablet report, sending...
I20260923 18:13:28.197376 9261 ts_manager.cc:194] Registered new tserver with Master: 1b28b788863641d884c16c6e41fc03fb (127.27.75.193:44461)
I20260923 18:13:28.198501 9261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:53515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:13:28.212179 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39307
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:28.322597 9457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:28.322867 9457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:28.322911 9457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:28.327533 9457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:28.327600 9457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:28.327662 9457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:13:28.333216 9457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:28.334266 9457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:28.335242 9457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:28.344918 9463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:28.344898 9462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:28.345026 9457 server_base.cc:1034] running on GCE node
W20260923 18:13:28.344892 9465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:28.345598 9457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:28.346222 9457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:28.347398 9457 hybrid_clock.cc:648] HybridClock initialized: now 1790187208347371 us; error 41 us; skew 500 ppm
Sep 23 18:13:28 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:28.350162 9457 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:13:28.351017 9457 webserver.cc:460] Webserver started at http://127.27.75.194:41755/ using document root <none> and password file <none>
I20260923 18:13:28.351536 9457 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:28.351594 9457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:28.351765 9457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:28.353830 9457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "ee3d75bbfec5558f0d0146c57452f8bd"
server_key_iv: "e580a9e5fe4701bd5e74280bfa82cf91"
server_key_version: "encryptionkey@0"
I20260923 18:13:28.354350 9457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "ee3d75bbfec5558f0d0146c57452f8bd"
server_key_iv: "e580a9e5fe4701bd5e74280bfa82cf91"
server_key_version: "encryptionkey@0"
I20260923 18:13:28.358134 9457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:13:28.360713 9472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:28.362066 9457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:13:28.362214 9457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "ee3d75bbfec5558f0d0146c57452f8bd"
server_key_iv: "e580a9e5fe4701bd5e74280bfa82cf91"
server_key_version: "encryptionkey@0"
I20260923 18:13:28.362334 9457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:28.382066 9457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:28.385217 9457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:28.385392 9457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:28.385958 9457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:28.386756 9457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:28.386801 9457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:28.386845 9457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:28.386873 9457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:28.395252 9457 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:43811
I20260923 18:13:28.395305 9585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:43811 every 8 connection(s)
I20260923 18:13:28.396145 9457 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:13:28.398674 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 9457
I20260923 18:13:28.398787 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:13:28.399070 27951 external_mini_cluster.cc:1428] Setting key c4175f91d4ef7fa5272b6cef5e78d297
Sep 23 18:13:28 dist-test-slave-wsx5 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:13:28.410182 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.397974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:60099 (local address 127.27.75.254:39307)
0923 18:13:28.398202 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:28.398206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:28.399077 (+ 871us) server_negotiation.cc:407] Connection header received
0923 18:13:28.400504 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:28.400508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:28.400559 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:28.400631 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:28.401871 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.402395 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.403529 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.403747 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.403835 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.406495 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:28.406498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:28.406500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:28.406528 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:28.408004 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:28.408556 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:28.408559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:28.408561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:28.408639 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:28.408982 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:28.408986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:28.408988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:28.409233 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:28.409324 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:28.409863 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 18:13:28.410010 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
I20260923 18:13:28.411120 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.398365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:60099)
0923 18:13:28.398891 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:28.398937 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:28.400311 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:28.400817 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:28.400825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:28.401234 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:28.401734 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.401749 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.402522 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.402524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.403408 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.403417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.403891 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.403895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.404065 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.404660 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:28.404676 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:28.406326 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:28.408128 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:28.408135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:28.408153 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:28.408451 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:28.408745 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:28.408748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:28.408750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:28.408890 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:28.409332 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:28.409338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:28.409620 (+ 282us) client_negotiation.cc:769] Sending connection context
0923 18:13:28.409798 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 18:13:28.410146 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":129,"threads_started":1}
I20260923 18:13:28.412271 9586 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39307
I20260923 18:13:28.412565 9586 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:28.413223 9586 heartbeater.cc:511] Master 127.27.75.254:39307 requested a full tablet report, sending...
I20260923 18:13:28.414381 9261 ts_manager.cc:194] Registered new tserver with Master: 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811)
I20260923 18:13:28.414960 9261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:60099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:13:28.428068 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39307
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:28.538760 9591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:28.539037 9591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:28.539081 9591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:28.543135 9591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:28.543205 9591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:28.543277 9591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:13:28.548682 9591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:28.549849 9591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:28.550935 9591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:28.560948 9599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:28.560981 9597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:28.560994 9596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:28.561188 9591 server_base.cc:1034] running on GCE node
I20260923 18:13:28.561780 9591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:28.562476 9591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:28.563678 9591 hybrid_clock.cc:648] HybridClock initialized: now 1790187208563651 us; error 37 us; skew 500 ppm
Sep 23 18:13:28 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:28.566922 9591 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:13:28.567916 9591 webserver.cc:460] Webserver started at http://127.27.75.195:33491/ using document root <none> and password file <none>
I20260923 18:13:28.568478 9591 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:28.568549 9591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:28.568758 9591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:28.570912 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "f9d427254ec04ecb91796e849092517d"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "5d20f082dbf67b6e59ec439a4b0a04c5"
server_key_iv: "3176cb584435ed5126bb2334feaae99c"
server_key_version: "encryptionkey@0"
I20260923 18:13:28.571486 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "f9d427254ec04ecb91796e849092517d"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "5d20f082dbf67b6e59ec439a4b0a04c5"
server_key_iv: "3176cb584435ed5126bb2334feaae99c"
server_key_version: "encryptionkey@0"
I20260923 18:13:28.575644 9591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:13:28.578382 9606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:28.579795 9591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:13:28.579941 9591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "f9d427254ec04ecb91796e849092517d"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "5d20f082dbf67b6e59ec439a4b0a04c5"
server_key_iv: "3176cb584435ed5126bb2334feaae99c"
server_key_version: "encryptionkey@0"
I20260923 18:13:28.580063 9591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:28.604092 9591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:28.607645 9591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:28.607856 9591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:28.608484 9591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:28.609412 9591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:28.609462 9591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:28.609521 9591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:28.609547 9591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:28.618350 9591 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:32787
I20260923 18:13:28.618381 9719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:32787 every 8 connection(s)
I20260923 18:13:28.619268 9591 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:13:28.624812 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 9591
I20260923 18:13:28.624943 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:13:28.625227 27951 external_mini_cluster.cc:1428] Setting key 770adaa8f1dc514473c669b061202eef
Sep 23 18:13:28 dist-test-slave-wsx5 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:28.633282 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.621050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:39461 (local address 127.27.75.254:39307)
0923 18:13:28.621223 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:28.621227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:28.622121 (+ 894us) server_negotiation.cc:407] Connection header received
0923 18:13:28.623619 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:28.623623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:28.623682 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:28.623756 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:28.624942 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.625628 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.626999 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.627294 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.627374 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.629953 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:28.629956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:28.629958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:28.629984 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:28.631183 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:28.631656 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:28.631659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:28.631660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:28.631731 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:28.632027 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:28.632030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:28.632031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:28.632209 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:28.632284 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:28.632966 (+ 682us) server_negotiation.cc:299] Negotiation successful
0923 18:13:28.633084 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:13:28.634083 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.621445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:39461)
0923 18:13:28.621948 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:28.622005 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:28.623402 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:28.623934 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:28.623941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:28.624293 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:28.624803 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.624816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.625781 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.625784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.626878 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.626888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.627506 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.627509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.627673 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.628258 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:28.628274 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:28.629787 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:28.631291 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:28.631296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:28.631308 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:28.631572 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:28.631796 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:28.631798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:28.631800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:28.631939 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:28.632322 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:28.632327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:28.632609 (+ 282us) client_negotiation.cc:769] Sending connection context
0923 18:13:28.632787 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 18:13:28.633167 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":107,"threads_started":1}
I20260923 18:13:28.635268 9720 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39307
I20260923 18:13:28.635519 9720 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:28.636034 9720 heartbeater.cc:511] Master 127.27.75.254:39307 requested a full tablet report, sending...
I20260923 18:13:28.637178 9261 ts_manager.cc:194] Registered new tserver with Master: f9d427254ec04ecb91796e849092517d (127.27.75.195:32787)
I20260923 18:13:28.637831 9261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:39461
I20260923 18:13:28.639004 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:13:28.648324 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.640907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.27.75.254:39307)
0923 18:13:28.641035 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:28.641039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:28.641225 (+ 186us) server_negotiation.cc:407] Connection header received
0923 18:13:28.641353 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:28.641356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:28.641400 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:28.641458 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:28.642186 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.642702 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.643589 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.643814 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.643900 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.645309 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:28.645313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:28.645316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:28.645343 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:28.646589 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:28.647063 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:28.647065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:28.647067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:28.647157 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:28.647450 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:28.647452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:28.647453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:28.647688 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:28.647790 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:28.647988 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 18:13:28.648160 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 18:13:28.651360 9261 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:13:28.654495 9261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:13:28.668896 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.663252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:32787 (local address 127.0.0.1:36842)
0923 18:13:28.663570 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:28.663612 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:28.663742 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:28.664471 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:28.664477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:28.664514 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:28.664672 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.664680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.666095 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.666098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.667311 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.667323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.668366 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.668370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.668475 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.668514 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 18:13:28.668621 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 18:13:28.668749 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":53,"thread_start_us":106,"threads_started":1}
I20260923 18:13:28.668905 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.663417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.27.75.195:32787)
0923 18:13:28.663804 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:28.663816 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:28.663838 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:13:28.663913 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:28.663918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:28.664131 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:28.664278 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:28.664804 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.665968 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.667449 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.668282 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.668372 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.668533 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:28.668657 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 18:13:28.668750 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":185,"threads_started":1}
I20260923 18:13:28.669854 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.664031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:43811 (local address 127.0.0.1:48576)
0923 18:13:28.664806 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:28.664819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:28.664905 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:28.665397 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:28.665400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:28.665417 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:28.665534 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.665541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.666907 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.666910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.668204 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.668215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.669451 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.669455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.669543 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.669567 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:13:28.669648 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 18:13:28.669714 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":6,"thread_start_us":88,"threads_started":1}
I20260923 18:13:28.669952 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.664167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.27.75.194:43811)
0923 18:13:28.664623 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:28.664634 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:28.665007 (+ 373us) server_negotiation.cc:407] Connection header received
0923 18:13:28.665076 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:28.665080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:28.665279 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:28.665413 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:28.665643 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.666713 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.668330 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.669169 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.669262 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.669390 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:28.669703 (+ 313us) server_negotiation.cc:299] Negotiation successful
0923 18:13:28.669807 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":120,"threads_started":1}
I20260923 18:13:28.670605 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.664353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:44461 (local address 127.0.0.1:44376)
0923 18:13:28.664886 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:28.664899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:28.664974 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:28.665821 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:28.665824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:28.665840 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:28.665940 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.665945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.667460 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.667464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.668385 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.668396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.670209 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.670212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.670306 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.670328 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:13:28.670401 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 18:13:28.670472 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":46,"thread_start_us":70,"threads_started":1}
I20260923 18:13:28.671278 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.664531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.27.75.193:44461)
0923 18:13:28.665230 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:28.665243 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:28.665265 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:13:28.665339 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:28.665344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:28.665524 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:28.665656 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:28.666054 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.667326 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.668530 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.669769 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.669862 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.670979 (+ 1117us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:28.671031 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 18:13:28.671128 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":87,"threads_started":1}
I20260923 18:13:28.672046 9654 tablet_service.cc:1467] Processing CreateTablet for tablet 8cebb3c24c224f3ca59c029285dc7238 (DEFAULT_TABLE table=test-table [id=fd069076627443e1941aba4096617f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:28.673277 9654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cebb3c24c224f3ca59c029285dc7238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:28.672881 9520 tablet_service.cc:1467] Processing CreateTablet for tablet 8cebb3c24c224f3ca59c029285dc7238 (DEFAULT_TABLE table=test-table [id=fd069076627443e1941aba4096617f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:28.674005 9520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cebb3c24c224f3ca59c029285dc7238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:28.674027 9386 tablet_service.cc:1467] Processing CreateTablet for tablet 8cebb3c24c224f3ca59c029285dc7238 (DEFAULT_TABLE table=test-table [id=fd069076627443e1941aba4096617f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:28.674954 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cebb3c24c224f3ca59c029285dc7238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:28.679921 9737 tablet_bootstrap.cc:492] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Bootstrap starting.
I20260923 18:13:28.681537 9739 tablet_bootstrap.cc:492] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Bootstrap starting.
I20260923 18:13:28.683001 9737 tablet_bootstrap.cc:654] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:28.683454 9739 tablet_bootstrap.cc:654] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:28.683784 9737 log.cc:824] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:28.684561 9739 log.cc:824] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:28.685142 9738 tablet_bootstrap.cc:492] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: Bootstrap starting.
I20260923 18:13:28.685994 9737 tablet_bootstrap.cc:492] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: No bootstrap required, opened a new log
I20260923 18:13:28.686198 9737 ts_tablet_manager.cc:1397] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 18:13:28.686528 9739 tablet_bootstrap.cc:492] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: No bootstrap required, opened a new log
I20260923 18:13:28.686720 9739 ts_tablet_manager.cc:1397] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 18:13:28.687508 9738 tablet_bootstrap.cc:654] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:28.688328 9738 log.cc:824] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:28.690212 9737 raft_consensus.cc:348] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:28.690488 9738 tablet_bootstrap.cc:492] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: No bootstrap required, opened a new log
I20260923 18:13:28.690516 9737 raft_consensus.cc:374] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:28.690578 9737 raft_consensus.cc:729] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd72cbcd08a4ad2846c4a62dc96ff35, State: Initialized, Role: FOLLOWER
I20260923 18:13:28.690671 9738 ts_tablet_manager.cc:1397] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260923 18:13:28.690866 9739 raft_consensus.cc:348] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:28.691087 9737 consensus_queue.cc:260] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:28.691223 9739 raft_consensus.cc:374] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:28.691288 9739 raft_consensus.cc:729] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b28b788863641d884c16c6e41fc03fb, State: Initialized, Role: FOLLOWER
I20260923 18:13:28.691800 9586 heartbeater.cc:503] Master 127.27.75.254:39307 was elected leader, sending a full tablet report...
I20260923 18:13:28.692142 9737 ts_tablet_manager.cc:1428] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:13:28.692358 9739 consensus_queue.cc:260] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:28.693077 9452 heartbeater.cc:503] Master 127.27.75.254:39307 was elected leader, sending a full tablet report...
I20260923 18:13:28.694378 9739 ts_tablet_manager.cc:1428] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260923 18:13:28.694214 9738 raft_consensus.cc:348] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:28.694480 9738 raft_consensus.cc:374] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:28.694538 9738 raft_consensus.cc:729] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d427254ec04ecb91796e849092517d, State: Initialized, Role: FOLLOWER
I20260923 18:13:28.695009 9738 consensus_queue.cc:260] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:28.695845 9738 ts_tablet_manager.cc:1428] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260923 18:13:28.696169 9720 heartbeater.cc:503] Master 127.27.75.254:39307 was elected leader, sending a full tablet report...
W20260923 18:13:28.870265 9721 tablet.cc:2369] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 18:13:28.897341 9587 tablet.cc:2369] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:13:28.921753 9745 raft_consensus.cc:482] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:13:28.921978 9745 raft_consensus.cc:504] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:28.923202 9745 leader_election.cc:288] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811), 1b28b788863641d884c16c6e41fc03fb (127.27.75.193:44461)
I20260923 18:13:28.927057 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.923478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:43811 (local address 127.27.75.195:60321)
0923 18:13:28.923617 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:28.923634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:28.923751 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:28.924063 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:28.924066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:28.924089 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:28.924226 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.924232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.925198 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.925202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.926007 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.926016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.926643 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.926647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.926754 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.926772 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 18:13:28.926836 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 18:13:28.926912 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 18:13:28.927057 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.923571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:60321 (local address 127.27.75.194:43811)
0923 18:13:28.923696 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:28.923701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:28.923718 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:13:28.923863 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:28.923865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:28.923917 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:28.923982 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:28.924328 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.925066 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.926130 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.926520 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.926632 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.926689 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:28.926860 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 18:13:28.926934 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 18:13:28.927302 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.923773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:44461 (local address 127.27.75.195:37255)
0923 18:13:28.924090 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:28.924102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:28.924214 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:28.924449 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:28.924452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:28.924469 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:28.924583 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.924588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.925512 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.925515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.926283 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:28.926292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.926899 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.926903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:28.927002 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.927025 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:13:28.927091 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 18:13:28.927159 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260923 18:13:28.927289 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.923951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:37255 (local address 127.27.75.193:44461)
0923 18:13:28.924088 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:28.924093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:28.924192 (+ 99us) server_negotiation.cc:407] Connection header received
0923 18:13:28.924304 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:28.924307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:28.924354 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:28.924430 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:28.924680 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.925403 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.926409 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.926793 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.926899 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.926951 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:28.927116 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 18:13:28.927185 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 18:13:28.927754 9540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cebb3c24c224f3ca59c029285dc7238" candidate_uuid: "f9d427254ec04ecb91796e849092517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" is_pre_election: true
I20260923 18:13:28.928097 9540 raft_consensus.cc:2454] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d427254ec04ecb91796e849092517d in term 0.
I20260923 18:13:28.928141 9406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cebb3c24c224f3ca59c029285dc7238" candidate_uuid: "f9d427254ec04ecb91796e849092517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b28b788863641d884c16c6e41fc03fb" is_pre_election: true
W20260923 18:13:28.928423 9453 tablet.cc:2369] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:13:28.928423 9406 raft_consensus.cc:2454] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d427254ec04ecb91796e849092517d in term 0.
I20260923 18:13:28.928593 9610 leader_election.cc:302] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dd72cbcd08a4ad2846c4a62dc96ff35, f9d427254ec04ecb91796e849092517d; no voters:
I20260923 18:13:28.928905 9745 raft_consensus.cc:2781] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:13:28.929003 9745 raft_consensus.cc:482] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:13:28.929046 9745 raft_consensus.cc:3037] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:28.930089 9745 raft_consensus.cc:504] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:28.930428 9745 leader_election.cc:288] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [CANDIDATE]: Term 1 election: Requested vote from peers 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811), 1b28b788863641d884c16c6e41fc03fb (127.27.75.193:44461)
I20260923 18:13:28.930917 9540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cebb3c24c224f3ca59c029285dc7238" candidate_uuid: "f9d427254ec04ecb91796e849092517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35"
I20260923 18:13:28.931066 9540 raft_consensus.cc:3037] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:28.931083 9406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cebb3c24c224f3ca59c029285dc7238" candidate_uuid: "f9d427254ec04ecb91796e849092517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b28b788863641d884c16c6e41fc03fb"
I20260923 18:13:28.931212 9406 raft_consensus.cc:3037] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:28.932163 9406 raft_consensus.cc:2454] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d427254ec04ecb91796e849092517d in term 1.
I20260923 18:13:28.932161 9540 raft_consensus.cc:2454] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9d427254ec04ecb91796e849092517d in term 1.
I20260923 18:13:28.932446 9610 leader_election.cc:302] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dd72cbcd08a4ad2846c4a62dc96ff35, f9d427254ec04ecb91796e849092517d; no voters:
I20260923 18:13:28.932618 9745 raft_consensus.cc:2781] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:13:28.932842 9745 raft_consensus.cc:686] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 1 LEADER]: Becoming Leader. State: Replica: f9d427254ec04ecb91796e849092517d, State: Running, Role: LEADER
I20260923 18:13:28.933223 9745 consensus_queue.cc:237] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:28.936816 9259 catalog_manager.cc:5495] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d reported cstate change: term changed from 0 to 1, leader changed from <none> to f9d427254ec04ecb91796e849092517d (127.27.75.195). New cstate: current_term: 1 leader_uuid: "f9d427254ec04ecb91796e849092517d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } health_report { overall_health: UNKNOWN } } }
I20260923 18:13:28.966847 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:28.963190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.27.75.195:32787)
0923 18:13:28.963304 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:28.963308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:28.963320 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:13:28.963457 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:28.963460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:28.963514 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:28.963603 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:13:28.963872 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.964361 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.965269 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:28.965496 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:28.965587 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:28.965936 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:13:28.966405 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:13:28.966521 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:28.966639 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 18:13:28.966695 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 18:13:28.973181 9406 raft_consensus.cc:1261] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Refusing update from remote peer f9d427254ec04ecb91796e849092517d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:13:28.973299 9540 raft_consensus.cc:1261] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 1 FOLLOWER]: Refusing update from remote peer f9d427254ec04ecb91796e849092517d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:13:28.973924 9745 consensus_queue.cc:1035] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:13:28.974143 9747 consensus_queue.cc:1035] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:13:28.985428 9751 mvcc.cc:205] Tried to move back new op lower bound from 7332606807945687040 to 7332606807793164288. Current Snapshot: MvccSnapshot[applied={T|T < 7332606807945687040}]
I20260923 18:13:28.990675 9261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:13:28.990854 9261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:13:28.993327 9261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Sending DeleteTablet for 3 replicas of tablet 8cebb3c24c224f3ca59c029285dc7238
I20260923 18:13:28.994313 9654 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cebb3c24c224f3ca59c029285dc7238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:28 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:36842
I20260923 18:13:28.994346 9386 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cebb3c24c224f3ca59c029285dc7238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:28 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:44376
I20260923 18:13:28.994349 9520 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cebb3c24c224f3ca59c029285dc7238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:28 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:48576
I20260923 18:13:28.994858 9761 tablet_replica.cc:330] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: stopping tablet replica
I20260923 18:13:28.994901 9760 tablet_replica.cc:330] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: stopping tablet replica
I20260923 18:13:28.994936 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 9323
I20260923 18:13:28.995232 9760 raft_consensus.cc:2229] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:13:28.995594 9760 raft_consensus.cc:2258] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:13:28.998039 9760 ts_tablet_manager.cc:1905] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:13:29.000258 9760 ts_tablet_manager.cc:1918] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:13:29.000362 9760 log.cc:1196] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/8cebb3c24c224f3ca59c029285dc7238
I20260923 18:13:29.000660 9760 ts_tablet_manager.cc:1939] T 8cebb3c24c224f3ca59c029285dc7238 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Deleting consensus metadata
I20260923 18:13:29.000797 9762 tablet_replica.cc:330] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: stopping tablet replica
I20260923 18:13:29.001156 9762 raft_consensus.cc:2229] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:13:29.001740 9762 raft_consensus.cc:2258] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 18:13:29.002146 9248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:44461 (error 108)
I20260923 18:13:29.002733 9248 catalog_manager.cc:4841] TS 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811): tablet 8cebb3c24c224f3ca59c029285dc7238 (table test-table [id=fd069076627443e1941aba4096617f61]) successfully deleted
I20260923 18:13:29.003374 9762 ts_tablet_manager.cc:1905] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:13:29.003510 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 9457
I20260923 18:13:29.003858 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.003562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:44461 (local address 127.0.0.1:44380)
0923 18:13:29.003690 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:29.003773 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:44461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 18:13:29.004812 9248 catalog_manager.cc:4584] TS 1b28b788863641d884c16c6e41fc03fb (127.27.75.193:44461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8cebb3c24c224f3ca59c029285dc7238: Network error: Client connection negotiation failed: client connection to 127.27.75.193:44461: connect: Connection refused (error 111)
I20260923 18:13:29.006397 9762 ts_tablet_manager.cc:1918] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:13:29.006505 9762 log.cc:1196] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/8cebb3c24c224f3ca59c029285dc7238
I20260923 18:13:29.006788 9762 ts_tablet_manager.cc:1939] T 8cebb3c24c224f3ca59c029285dc7238 P f9d427254ec04ecb91796e849092517d: Deleting consensus metadata
I20260923 18:13:29.007905 9248 catalog_manager.cc:4841] TS f9d427254ec04ecb91796e849092517d (127.27.75.195:32787): tablet 8cebb3c24c224f3ca59c029285dc7238 (table test-table [id=fd069076627443e1941aba4096617f61]) successfully deleted
I20260923 18:13:29.010797 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 9591
I20260923 18:13:29.016181 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 9229
I20260923 18:13:29.022399 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39307
--webserver_interface=127.27.75.254
--webserver_port=35323
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:29.138708 9764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:29.139011 9764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:29.139055 9764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:29.143507 9764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:13:29.143590 9764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:29.143610 9764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:29.143626 9764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:13:29.143640 9764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:13:29.149395 9764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:39307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=35323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:29.150532 9764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:29.151643 9764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:29.165048 9770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:29.165181 9771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:29.165238 9773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:29.165417 9764 server_base.cc:1034] running on GCE node
I20260923 18:13:29.166013 9764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:29.167237 9764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:29.168417 9764 hybrid_clock.cc:648] HybridClock initialized: now 1790187209168366 us; error 67 us; skew 500 ppm
Sep 23 18:13:29 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187209, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:29.171420 9764 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:13:29.172354 9764 webserver.cc:460] Webserver started at http://127.27.75.254:35323/ using document root <none> and password file <none>
I20260923 18:13:29.172998 9764 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:29.173081 9764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:29.177450 9764 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 18:13:29.179901 9780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:29.181428 9764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:13:29.181552 9764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "38f41e55c743454a9a9d02f3805702f4"
format_stamp: "Formatted at 2026-09-23 18:13:27 on dist-test-slave-wsx5"
server_key: "592d78a1e671c48c6ebc8a8f046502e1"
server_key_iv: "649b3339777494cd204735e3bf1e3860"
server_key_version: "encryptionkey@0"
I20260923 18:13:29.182202 9764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:29.205737 9764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:29.209069 9764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:29.209282 9764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:29.216476 9764 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:39307
I20260923 18:13:29.216774 9832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:39307 every 8 connection(s)
I20260923 18:13:29.217680 9764 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:13:29.219753 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 9764
I20260923 18:13:29.220224 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:44461
--local_ip_for_outbound_sockets=127.27.75.193
--tserver_master_addrs=127.27.75.254:39307
--webserver_port=35903
--webserver_interface=127.27.75.193
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:13:29.222857 9833 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:13:29.224692 9833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Bootstrap starting.
I20260923 18:13:29.241912 9833 log.cc:824] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:29.250962 9833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:13:29.251361 9833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Bootstrap complete.
I20260923 18:13:29.255355 9833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:29.255779 9833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f41e55c743454a9a9d02f3805702f4, State: Initialized, Role: FOLLOWER
I20260923 18:13:29.256301 9833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:29.256422 9833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:13:29.256493 9833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:13:29.256584 9833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:13:29.258603 9833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:29.258958 9833 leader_election.cc:302] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38f41e55c743454a9a9d02f3805702f4; no voters:
I20260923 18:13:29.259289 9833 leader_election.cc:288] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 18:13:29.259399 9837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 18:13:29.259618 9837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 2 LEADER]: Becoming Leader. State: Replica: 38f41e55c743454a9a9d02f3805702f4, State: Running, Role: LEADER
I20260923 18:13:29.259920 9837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:29.260592 9833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:13:29.260942 9837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f41e55c743454a9a9d02f3805702f4. Latest consensus state: current_term: 2 leader_uuid: "38f41e55c743454a9a9d02f3805702f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } } }
I20260923 18:13:29.261056 9837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:29.261498 9838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "38f41e55c743454a9a9d02f3805702f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } } }
I20260923 18:13:29.261667 9838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:29.262028 9841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:13:29.266983 9841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:13:29.267622 9841 catalog_manager.cc:1259] Loaded cluster ID: f49217422d81469d9e4bf41aa928b8b0
I20260923 18:13:29.267724 9841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:13:29.269109 9841 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 18:13:29.273895 9841 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 18:13:29.279630 9841 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:13:29.280495 9841 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Loaded TSK: 0
I20260923 18:13:29.281837 9841 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:13:29.338980 9835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:29.339327 9835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:29.339394 9835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:29.343885 9835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:29.343993 9835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:29.344082 9835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:13:29.350455 9835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:44461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=35903
--tserver_master_addrs=127.27.75.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:29.351784 9835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:29.353013 9835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:29.365654 9863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:29.365658 9860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:29.366251 9835 server_base.cc:1034] running on GCE node
W20260923 18:13:29.366734 9861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:29.367292 9835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:29.368153 9835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:29.369318 9835 hybrid_clock.cc:648] HybridClock initialized: now 1790187209369302 us; error 31 us; skew 500 ppm
Sep 23 18:13:29 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187209, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:29.387562 9835 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:13:29.388675 9835 webserver.cc:460] Webserver started at http://127.27.75.193:35903/ using document root <none> and password file <none>
I20260923 18:13:29.389323 9835 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:29.389428 9835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:29.393519 9835 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260923 18:13:29.395661 9870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:29.396903 9835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:13:29.397006 9835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "1b28b788863641d884c16c6e41fc03fb"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "b1adbf168f0cc5818fa287b94c98a2bc"
server_key_iv: "cfc170784cfc775ad5063fbabf6bc96c"
server_key_version: "encryptionkey@0"
I20260923 18:13:29.397622 9835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:29.409623 9835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:29.412719 9835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:29.412894 9835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:29.413535 9835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:29.414633 9877 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 18:13:29.418751 9835 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 18:13:29.418812 9835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260923 18:13:29.418862 9835 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 18:13:29.420483 9835 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 18:13:29.420538 9835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260923 18:13:29.420819 9877 tablet_bootstrap.cc:492] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Bootstrap starting.
I20260923 18:13:29.430435 9835 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:44461
I20260923 18:13:29.430581 9984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:44461 every 8 connection(s)
I20260923 18:13:29.431752 9835 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:13:29.438694 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 9835
I20260923 18:13:29.439229 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:43811
--local_ip_for_outbound_sockets=127.27.75.194
--tserver_master_addrs=127.27.75.254:39307
--webserver_port=41755
--webserver_interface=127.27.75.194
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:13:29.441460 9877 log.cc:824] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:13:29 dist-test-slave-wsx5 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187209, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:29.449944 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.433979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:33687 (local address 127.27.75.254:39307)
0923 18:13:29.434453 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:29.434471 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:29.434909 (+ 438us) server_negotiation.cc:407] Connection header received
0923 18:13:29.436036 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:29.436049 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:29.436369 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:29.436698 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:29.438149 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.439555 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.441488 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.442086 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.442267 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.445045 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:29.445051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:29.445064 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:29.445098 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:29.446786 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:29.447319 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:29.447325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:29.447329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:29.447453 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:29.447772 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:29.447775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:29.447776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:29.448139 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:29.448261 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:29.448780 (+ 519us) server_negotiation.cc:299] Negotiation successful
0923 18:13:29.449050 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":111,"threads_started":1}
I20260923 18:13:29.449958 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.434025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:33687)
0923 18:13:29.434721 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:29.434766 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:29.435603 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:29.436905 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:29.436916 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:29.437445 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:29.437951 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.437967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.439727 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.439731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.441313 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.441327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.442248 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.442252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.442451 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.443022 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:29.443039 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:29.444692 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:29.446907 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:29.446912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:29.446926 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:29.447205 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:29.447548 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:29.447552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:29.447554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:29.447684 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:29.448246 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:29.448252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:29.448479 (+ 227us) client_negotiation.cc:769] Sending connection context
0923 18:13:29.448650 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 18:13:29.448999 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":150,"threads_started":1}
I20260923 18:13:29.450769 9877 tablet_bootstrap.cc:492] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:13:29.451195 9877 tablet_bootstrap.cc:492] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Bootstrap complete.
I20260923 18:13:29.452478 9877 ts_tablet_manager.cc:1397] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.011s
I20260923 18:13:29.452484 9985 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39307
I20260923 18:13:29.452742 9985 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:29.453274 9985 heartbeater.cc:511] Master 127.27.75.254:39307 requested a full tablet report, sending...
I20260923 18:13:29.455600 9793 ts_manager.cc:194] Registered new tserver with Master: 1b28b788863641d884c16c6e41fc03fb (127.27.75.193:44461)
I20260923 18:13:29.455538 9877 raft_consensus.cc:348] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:29.456076 9877 raft_consensus.cc:729] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b28b788863641d884c16c6e41fc03fb, State: Initialized, Role: FOLLOWER
I20260923 18:13:29.456568 9877 consensus_queue.cc:260] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:29.457653 9877 ts_tablet_manager.cc:1428] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260923 18:13:29.459265 9793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:33687
I20260923 18:13:29.461930 9985 heartbeater.cc:503] Master 127.27.75.254:39307 was elected leader, sending a full tablet report...
Sep 23 18:13:29 dist-test-slave-wsx5 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187209, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:13:29.470386 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.458669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:44461 (local address 127.0.0.1:44390)
0923 18:13:29.459269 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:29.459324 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:29.459458 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:29.460318 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:29.460323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:29.460832 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:29.460969 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.460977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.462592 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.462595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.463378 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.463389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.464138 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.464142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.464259 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.464792 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:29.464805 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:29.466499 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:29.468505 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:29.468516 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:29.468522 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:29.468777 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:29.469086 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:29.469089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:29.469091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:29.469224 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:29.469694 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:29.469700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:29.469830 (+ 130us) client_negotiation.cc:769] Sending connection context
0923 18:13:29.469952 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 18:13:29.470193 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":109,"threads_started":1}
I20260923 18:13:29.470386 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.458779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.27.75.193:44461)
0923 18:13:29.459724 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:29.459735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:29.459756 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:13:29.459826 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:29.459831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:29.460031 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:29.460152 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:29.461142 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.462457 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.463507 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.464021 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.464120 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.466690 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:29.466701 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:29.466713 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:29.466737 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:29.468381 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:29.468899 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:29.468904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:29.468905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:29.468992 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:29.469310 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:29.469314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:29.469316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:29.469582 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:29.469705 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:29.470051 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 18:13:29.470222 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":87,"threads_started":1}
I20260923 18:13:29.471393 9919 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cebb3c24c224f3ca59c029285dc7238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:28 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:44390
I20260923 18:13:29.472265 9994 tablet_replica.cc:330] stopping tablet replica
I20260923 18:13:29.472590 9994 raft_consensus.cc:2229] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:13:29.472868 9994 raft_consensus.cc:2258] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:13:29.474120 9994 ts_tablet_manager.cc:1905] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:13:29.478911 9994 ts_tablet_manager.cc:1918] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:13:29.479040 9994 log.cc:1196] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal/wals/8cebb3c24c224f3ca59c029285dc7238
I20260923 18:13:29.479293 9994 ts_tablet_manager.cc:1939] T 8cebb3c24c224f3ca59c029285dc7238 P 1b28b788863641d884c16c6e41fc03fb: Deleting consensus metadata
I20260923 18:13:29.481472 9784 catalog_manager.cc:4841] TS 1b28b788863641d884c16c6e41fc03fb (127.27.75.193:44461): tablet 8cebb3c24c224f3ca59c029285dc7238 (table test-table [id=fd069076627443e1941aba4096617f61]) successfully deleted
W20260923 18:13:29.564569 9989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:29.564854 9989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:29.564898 9989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:29.569146 9989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:29.569229 9989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:29.569298 9989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:13:29.574793 9989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:43811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=41755
--tserver_master_addrs=127.27.75.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:29.575889 9989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:29.576929 9989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:29.586957 9999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:29.587028 9989 server_base.cc:1034] running on GCE node
W20260923 18:13:29.586934 10002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:29.586973 10000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:29.587643 9989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:29.588312 9989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:29.589478 9989 hybrid_clock.cc:648] HybridClock initialized: now 1790187209589446 us; error 49 us; skew 500 ppm
Sep 23 18:13:29 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187209, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:29.592253 9989 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:13:29.593312 9989 webserver.cc:460] Webserver started at http://127.27.75.194:41755/ using document root <none> and password file <none>
I20260923 18:13:29.593870 9989 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:29.593945 9989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:29.598059 9989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:13:29.600350 10009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:29.601925 9989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:13:29.602075 9989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "ee3d75bbfec5558f0d0146c57452f8bd"
server_key_iv: "e580a9e5fe4701bd5e74280bfa82cf91"
server_key_version: "encryptionkey@0"
I20260923 18:13:29.602671 9989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:29.628402 9989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:29.631707 9989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:29.631868 9989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:29.632423 9989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:29.633292 9989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:29.633342 9989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:29.633383 9989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:29.633402 9989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:29.641948 9989 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:43811
I20260923 18:13:29.641978 10122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:43811 every 8 connection(s)
I20260923 18:13:29.642951 9989 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:13:29.647240 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 9989
I20260923 18:13:29.647605 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:32787
--local_ip_for_outbound_sockets=127.27.75.195
--tserver_master_addrs=127.27.75.254:39307
--webserver_port=33491
--webserver_interface=127.27.75.195
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:13:29 dist-test-slave-wsx5 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187209, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:29.657284 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.644748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:54653 (local address 127.27.75.254:39307)
0923 18:13:29.644943 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:29.644948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:29.645758 (+ 810us) server_negotiation.cc:407] Connection header received
0923 18:13:29.647090 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:29.647093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:29.647154 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:29.647226 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:29.648470 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.649279 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.650459 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.650804 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.650976 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.653602 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:29.653606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:29.653610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:29.653644 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:29.655178 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:29.655677 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:29.655681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:29.655683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:29.655779 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:29.656078 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:29.656081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:29.656083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:29.656325 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:29.656430 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:29.656912 (+ 482us) server_negotiation.cc:299] Negotiation successful
0923 18:13:29.657072 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 18:13:29.658051 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.645105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:54653)
0923 18:13:29.645609 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:29.645650 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:29.646833 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:29.647435 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:29.647442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:29.647840 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:29.648322 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.648335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.649410 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.649413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.650329 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.650338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.651047 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.651051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.651217 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.651899 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:29.651916 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:29.653407 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:29.655291 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:29.655296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:29.655310 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:29.655575 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:29.655863 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:29.655865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:29.655866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:29.655996 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:29.656413 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:29.656418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:29.656679 (+ 261us) client_negotiation.cc:769] Sending connection context
0923 18:13:29.656836 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 18:13:29.657104 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":115,"threads_started":1}
I20260923 18:13:29.659219 10123 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39307
I20260923 18:13:29.659490 10123 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:29.660090 10123 heartbeater.cc:511] Master 127.27.75.254:39307 requested a full tablet report, sending...
I20260923 18:13:29.661325 9793 ts_manager.cc:194] Registered new tserver with Master: 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811)
I20260923 18:13:29.661989 9793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:54653
W20260923 18:13:29.768348 10126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:29.768633 10126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:29.768677 10126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:29.772853 10126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:29.772951 10126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:29.773013 10126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:13:29.778960 10126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:32787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=33491
--tserver_master_addrs=127.27.75.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:29.780103 10126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:29.781145 10126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:29.791285 10134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:29.791276 10131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:29.791271 10132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:29.791364 10126 server_base.cc:1034] running on GCE node
I20260923 18:13:29.791982 10126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:29.792640 10126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:29.793818 10126 hybrid_clock.cc:648] HybridClock initialized: now 1790187209793784 us; error 49 us; skew 500 ppm
Sep 23 18:13:29 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187209, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:29.796814 10126 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:13:29.797818 10126 webserver.cc:460] Webserver started at http://127.27.75.195:33491/ using document root <none> and password file <none>
I20260923 18:13:29.798388 10126 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:29.798471 10126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:29.802563 10126 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:13:29.804829 10141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:29.806414 10126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:13:29.806555 10126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "f9d427254ec04ecb91796e849092517d"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "5d20f082dbf67b6e59ec439a4b0a04c5"
server_key_iv: "3176cb584435ed5126bb2334feaae99c"
server_key_version: "encryptionkey@0"
I20260923 18:13:29.807137 10126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:29.824991 10126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:29.829032 10126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:29.829250 10126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:29.829860 10126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:29.830705 10126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:29.830752 10126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:29.830794 10126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:29.830818 10126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:29.839298 10126 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:32787
I20260923 18:13:29.839363 10254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:32787 every 8 connection(s)
I20260923 18:13:29.840348 10126 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:13:29.844436 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 10126
Sep 23 18:13:29 dist-test-slave-wsx5 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187209, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:29.853600 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.845105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.27.75.254:39307)
0923 18:13:29.845356 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:29.845360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:29.845507 (+ 147us) server_negotiation.cc:407] Connection header received
0923 18:13:29.845622 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:29.845625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:29.845669 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:29.845740 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:29.846465 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.847092 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.847945 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.848275 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.848354 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.849973 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:29.849977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:29.849979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:29.850011 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:29.851720 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:29.852172 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:29.852176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:29.852178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:29.852257 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:29.852524 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:29.852527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:29.852528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:29.852761 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:29.853003 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:29.853266 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 18:13:29.853405 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20260923 18:13:29.854799 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.842284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:34557 (local address 127.27.75.254:39307)
0923 18:13:29.842456 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:29.842461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:29.843406 (+ 945us) server_negotiation.cc:407] Connection header received
0923 18:13:29.844814 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:29.844819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:29.844867 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:29.844944 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:29.846329 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.846886 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.848213 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.848464 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.848558 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.850964 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:29.850968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:29.850971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:29.851021 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:29.852385 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:29.852989 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:29.852993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:29.852995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:29.853083 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:29.853533 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:29.853536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:29.853537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:29.853761 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:29.853837 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:29.854554 (+ 717us) server_negotiation.cc:299] Negotiation successful
0923 18:13:29.854665 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 18:13:29.855729 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.842692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:34557)
0923 18:13:29.843239 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:29.843287 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:29.844576 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:29.845219 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:29.845229 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:29.845645 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:29.846178 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.846193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.847033 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.847036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.848085 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.848097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.848609 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.848612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.848801 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.849336 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:29.849353 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:29.850769 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:29.852502 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:29.852509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:29.852528 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:29.852866 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:29.853215 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:29.853218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:29.853220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:29.853412 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:29.853926 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:29.853932 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:29.854306 (+ 374us) client_negotiation.cc:769] Sending connection context
0923 18:13:29.854516 (+ 210us) client_negotiation.cc:240] Negotiation successful
0923 18:13:29.854871 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":108,"threads_started":1}
I20260923 18:13:29.856864 10255 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39307
I20260923 18:13:29.857092 10255 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:29.857599 10255 heartbeater.cc:511] Master 127.27.75.254:39307 requested a full tablet report, sending...
I20260923 18:13:29.858567 9793 ts_manager.cc:194] Registered new tserver with Master: f9d427254ec04ecb91796e849092517d (127.27.75.195:32787)
I20260923 18:13:29.859138 9793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:34557
I20260923 18:13:29.867923 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:13:29.872534 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.868724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.27.75.254:39307)
0923 18:13:29.868918 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:29.868922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:29.869063 (+ 141us) server_negotiation.cc:407] Connection header received
0923 18:13:29.869267 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:29.869269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:29.869381 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:29.869475 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:13:29.869751 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.870295 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.871147 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.871359 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.871448 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.871751 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:13:29.872168 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:13:29.872273 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:29.872335 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 18:13:29.872396 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 18:13:29.873383 9793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:13:29.876709 9793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:13:29.885022 9919 tablet_service.cc:1467] Processing CreateTablet for tablet f15f8e6cbe7e49bf96914cb637161265 (DEFAULT_TABLE table=test-table [id=923987c451164931b894eb217584cf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:29.885771 9919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15f8e6cbe7e49bf96914cb637161265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:29.888643 9877 tablet_bootstrap.cc:492] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: Bootstrap starting.
I20260923 18:13:29.889026 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.883746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:43811 (local address 127.0.0.1:48592)
0923 18:13:29.883873 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:29.883888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:29.883964 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:29.884956 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:29.884959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:29.884988 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:29.885103 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.885110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.886696 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.886700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.887609 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.887619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.888554 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.888558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.888658 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.888718 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 18:13:29.888787 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 18:13:29.888875 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":5}
I20260923 18:13:29.889374 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.883924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.27.75.194:43811)
0923 18:13:29.884394 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:29.884406 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:29.884430 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:13:29.884520 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:29.884526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:29.884762 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:29.884961 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:29.885270 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.886570 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.887741 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.888417 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.888525 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.888850 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:29.888908 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 18:13:29.889017 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":174,"threads_started":1}
I20260923 18:13:29.889597 9877 tablet_bootstrap.cc:654] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:29.890463 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.884104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:32787 (local address 127.0.0.1:36860)
0923 18:13:29.884416 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:29.884430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:29.884526 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:29.885175 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:29.885178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:29.885197 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:29.885285 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.885290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.886957 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.886961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.888034 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:29.888045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.890011 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.890015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:29.890123 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.890149 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:13:29.890241 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 18:13:29.890322 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":80,"threads_started":1}
I20260923 18:13:29.890529 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:29.884249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36860 (local address 127.27.75.195:32787)
0923 18:13:29.884667 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:29.884677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:29.884695 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:13:29.884763 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:29.884768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:29.884935 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:29.885145 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:29.885380 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.886811 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.888162 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:29.889864 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:29.889952 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:29.890061 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:29.890301 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 18:13:29.890391 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":102,"threads_started":1}
I20260923 18:13:29.891351 9877 tablet_bootstrap.cc:492] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: No bootstrap required, opened a new log
I20260923 18:13:29.891443 9877 ts_tablet_manager.cc:1397] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 18:13:29.891932 9877 raft_consensus.cc:348] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:29.892084 9877 raft_consensus.cc:374] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:29.892131 9877 raft_consensus.cc:729] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b28b788863641d884c16c6e41fc03fb, State: Initialized, Role: FOLLOWER
I20260923 18:13:29.892279 9877 consensus_queue.cc:260] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:29.892632 9877 ts_tablet_manager.cc:1428] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 18:13:29.893620 10189 tablet_service.cc:1467] Processing CreateTablet for tablet f15f8e6cbe7e49bf96914cb637161265 (DEFAULT_TABLE table=test-table [id=923987c451164931b894eb217584cf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:29.893627 10057 tablet_service.cc:1467] Processing CreateTablet for tablet f15f8e6cbe7e49bf96914cb637161265 (DEFAULT_TABLE table=test-table [id=923987c451164931b894eb217584cf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:29.894634 10189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15f8e6cbe7e49bf96914cb637161265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:29.894642 10057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15f8e6cbe7e49bf96914cb637161265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:29.901360 10268 tablet_bootstrap.cc:492] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Bootstrap starting.
I20260923 18:13:29.901356 10269 tablet_bootstrap.cc:492] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: Bootstrap starting.
I20260923 18:13:29.903859 10268 tablet_bootstrap.cc:654] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:29.903939 10269 tablet_bootstrap.cc:654] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:29.904717 10269 log.cc:824] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:29.904717 10268 log.cc:824] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:29.906637 10268 tablet_bootstrap.cc:492] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: No bootstrap required, opened a new log
I20260923 18:13:29.906646 10269 tablet_bootstrap.cc:492] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: No bootstrap required, opened a new log
I20260923 18:13:29.906821 10268 ts_tablet_manager.cc:1397] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:13:29.906821 10269 ts_tablet_manager.cc:1397] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:13:29.909855 10268 raft_consensus.cc:348] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:29.909857 10269 raft_consensus.cc:348] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:29.910152 10268 raft_consensus.cc:374] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:29.910152 10269 raft_consensus.cc:374] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:29.910211 10269 raft_consensus.cc:729] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d427254ec04ecb91796e849092517d, State: Initialized, Role: FOLLOWER
I20260923 18:13:29.910221 10268 raft_consensus.cc:729] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd72cbcd08a4ad2846c4a62dc96ff35, State: Initialized, Role: FOLLOWER
I20260923 18:13:29.910637 10269 consensus_queue.cc:260] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:29.910645 10268 consensus_queue.cc:260] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:29.911523 10268 ts_tablet_manager.cc:1428] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 18:13:29.911525 10269 ts_tablet_manager.cc:1428] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 18:13:29.911878 10255 heartbeater.cc:503] Master 127.27.75.254:39307 was elected leader, sending a full tablet report...
I20260923 18:13:29.912333 10123 heartbeater.cc:503] Master 127.27.75.254:39307 was elected leader, sending a full tablet report...
W20260923 18:13:29.933326 9986 tablet.cc:2369] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:13:30.069293 10272 raft_consensus.cc:482] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:13:30.069546 10272 raft_consensus.cc:504] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:30.071012 10272 leader_election.cc:288] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d427254ec04ecb91796e849092517d (127.27.75.195:32787), 1b28b788863641d884c16c6e41fc03fb (127.27.75.193:44461)
I20260923 18:13:30.071509 10273 raft_consensus.cc:482] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:13:30.071746 10273 raft_consensus.cc:504] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:30.073036 10273 leader_election.cc:288] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811), 1b28b788863641d884c16c6e41fc03fb (127.27.75.193:44461)
I20260923 18:13:30.075778 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.071304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:32787 (local address 127.27.75.194:35511)
0923 18:13:30.071474 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:30.071493 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:30.071615 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:30.071996 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:30.072000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:30.072020 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:30.072134 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:30.072140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.073171 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.073175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:30.074356 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:30.074366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.075316 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.075319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:30.075432 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:30.075457 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 18:13:30.075525 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 18:13:30.075623 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 18:13:30.076261 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.071409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:35511 (local address 127.27.75.195:32787)
0923 18:13:30.071526 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:30.071531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:30.071556 (+ 25us) server_negotiation.cc:407] Connection header received
0923 18:13:30.071758 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:30.071762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:30.071811 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:30.071881 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:30.072280 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.073004 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.074489 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.075058 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.075202 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:30.075255 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:30.076045 (+ 790us) server_negotiation.cc:299] Negotiation successful
0923 18:13:30.076126 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:13:30.076958 10209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15f8e6cbe7e49bf96914cb637161265" candidate_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d427254ec04ecb91796e849092517d" is_pre_election: true
I20260923 18:13:30.077251 10209 raft_consensus.cc:2454] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd72cbcd08a4ad2846c4a62dc96ff35 in term 0.
I20260923 18:13:30.077392 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.073224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:43811 (local address 127.27.75.195:48975)
0923 18:13:30.073317 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:30.073331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:30.073414 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:30.073751 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:30.073755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:30.073773 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:30.073870 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:30.073876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.074745 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.074748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:30.075890 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:30.075901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.076975 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.076979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:30.077073 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:30.077097 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:13:30.077193 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 18:13:30.077270 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 18:13:30.077461 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.073308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:48975 (local address 127.27.75.194:43811)
0923 18:13:30.073406 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:30.073410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:30.073423 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:13:30.073497 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:30.073500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:30.073543 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:30.073617 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:30.073977 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.074622 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.076028 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.076622 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.076837 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:30.076959 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:30.077267 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 18:13:30.077349 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":39}
I20260923 18:13:30.077772 10013 leader_election.cc:302] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dd72cbcd08a4ad2846c4a62dc96ff35, f9d427254ec04ecb91796e849092517d; no voters:
I20260923 18:13:30.078043 10275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.071781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:56581 (local address 127.27.75.193:44461)
0923 18:13:30.072167 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:30.072172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:30.072186 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:13:30.072237 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:30.072239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:30.072292 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:30.072384 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:30.072662 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.073536 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.075288 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.075982 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.076068 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:30.076228 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:30.077815 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0923 18:13:30.077914 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":91,"threads_started":1}
I20260923 18:13:30.078342 10272 raft_consensus.cc:2781] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:13:30.078450 10272 raft_consensus.cc:482] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:13:30.078392 10077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15f8e6cbe7e49bf96914cb637161265" candidate_uuid: "f9d427254ec04ecb91796e849092517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" is_pre_election: true
I20260923 18:13:30.078506 10272 raft_consensus.cc:3037] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:30.078569 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.073473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:44461 (local address 127.27.75.195:37905)
0923 18:13:30.074146 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:30.074160 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:30.074246 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:30.074696 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:30.074699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:30.074714 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:30.074821 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:30.074826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.075851 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.075854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:30.077258 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:30.077266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.078198 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.078202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:30.078294 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:30.078317 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:13:30.078367 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 18:13:30.078438 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":118,"threads_started":1}
I20260923 18:13:30.078586 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.071652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:44461 (local address 127.27.75.194:56581)
0923 18:13:30.072029 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:30.072043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:30.072140 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:30.072416 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:30.072419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:30.072434 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:30.072549 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:30.072555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.073659 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.073662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:30.074932 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:30.074942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.076761 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.076765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:30.077667 (+ 902us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:30.077690 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:13:30.078201 (+ 511us) client_negotiation.cc:240] Negotiation successful
0923 18:13:30.078283 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":925,"thread_start_us":100,"threads_started":1}
I20260923 18:13:30.079051 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.073816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:37905 (local address 127.27.75.193:44461)
0923 18:13:30.074473 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:30.074477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:30.074489 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:13:30.074527 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:30.074530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:30.074570 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:30.074914 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:30.075015 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.075713 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.077386 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.078005 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.078749 (+ 744us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:30.078799 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:30.078834 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 18:13:30.078906 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":68,"threads_started":1}
I20260923 18:13:30.079147 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15f8e6cbe7e49bf96914cb637161265" candidate_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b28b788863641d884c16c6e41fc03fb" is_pre_election: true
I20260923 18:13:30.079459 9939 raft_consensus.cc:2454] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd72cbcd08a4ad2846c4a62dc96ff35 in term 0.
I20260923 18:13:30.079968 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15f8e6cbe7e49bf96914cb637161265" candidate_uuid: "f9d427254ec04ecb91796e849092517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b28b788863641d884c16c6e41fc03fb" is_pre_election: true
I20260923 18:13:30.080085 9939 raft_consensus.cc:2454] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9d427254ec04ecb91796e849092517d in term 0.
I20260923 18:13:30.080473 10145 leader_election.cc:302] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b28b788863641d884c16c6e41fc03fb, f9d427254ec04ecb91796e849092517d; no voters:
I20260923 18:13:30.080530 10272 raft_consensus.cc:504] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:30.080758 10273 raft_consensus.cc:2781] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:13:30.080802 10077 raft_consensus.cc:2379] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9d427254ec04ecb91796e849092517d in current term 1: Already voted for candidate 6dd72cbcd08a4ad2846c4a62dc96ff35 in this term.
I20260923 18:13:30.080925 10272 leader_election.cc:288] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [CANDIDATE]: Term 1 election: Requested vote from peers f9d427254ec04ecb91796e849092517d (127.27.75.195:32787), 1b28b788863641d884c16c6e41fc03fb (127.27.75.193:44461)
I20260923 18:13:30.080865 10273 raft_consensus.cc:482] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:13:30.081271 10209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15f8e6cbe7e49bf96914cb637161265" candidate_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d427254ec04ecb91796e849092517d"
I20260923 18:13:30.081374 10273 raft_consensus.cc:3037] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:30.081377 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15f8e6cbe7e49bf96914cb637161265" candidate_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b28b788863641d884c16c6e41fc03fb"
I20260923 18:13:30.081524 9939 raft_consensus.cc:3037] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:30.082640 9939 raft_consensus.cc:2454] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd72cbcd08a4ad2846c4a62dc96ff35 in term 1.
I20260923 18:13:30.082645 10273 raft_consensus.cc:504] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:30.082890 10209 raft_consensus.cc:2379] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6dd72cbcd08a4ad2846c4a62dc96ff35 in current term 1: Already voted for candidate f9d427254ec04ecb91796e849092517d in this term.
I20260923 18:13:30.083000 10013 leader_election.cc:302] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b28b788863641d884c16c6e41fc03fb, 6dd72cbcd08a4ad2846c4a62dc96ff35; no voters:
I20260923 18:13:30.083285 10273 leader_election.cc:288] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [CANDIDATE]: Term 1 election: Requested vote from peers 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811), 1b28b788863641d884c16c6e41fc03fb (127.27.75.193:44461)
I20260923 18:13:30.083442 9939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15f8e6cbe7e49bf96914cb637161265" candidate_uuid: "f9d427254ec04ecb91796e849092517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b28b788863641d884c16c6e41fc03fb"
I20260923 18:13:30.083617 9939 raft_consensus.cc:2379] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f9d427254ec04ecb91796e849092517d in current term 1: Already voted for candidate 6dd72cbcd08a4ad2846c4a62dc96ff35 in this term.
I20260923 18:13:30.083207 10272 raft_consensus.cc:2781] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:13:30.083372 10077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15f8e6cbe7e49bf96914cb637161265" candidate_uuid: "f9d427254ec04ecb91796e849092517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35"
I20260923 18:13:30.084208 10272 raft_consensus.cc:686] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 1 LEADER]: Becoming Leader. State: Replica: 6dd72cbcd08a4ad2846c4a62dc96ff35, State: Running, Role: LEADER
I20260923 18:13:30.084558 10272 consensus_queue.cc:237] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:30.087226 10145 leader_election.cc:302] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9d427254ec04ecb91796e849092517d; no voters: 1b28b788863641d884c16c6e41fc03fb, 6dd72cbcd08a4ad2846c4a62dc96ff35
I20260923 18:13:30.087499 10273 raft_consensus.cc:2726] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260923 18:13:30.088550 9797 catalog_manager.cc:5495] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194). New cstate: current_term: 1 leader_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } health_report { overall_health: UNKNOWN } } }
W20260923 18:13:30.091374 10256 tablet.cc:2369] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:13:30.133719 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.130175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.27.75.194:43811)
0923 18:13:30.130289 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:30.130293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:30.130305 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:13:30.130380 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:30.130383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:30.130445 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:30.130539 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:13:30.130805 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.131292 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.132189 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:30.132407 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:30.132488 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:30.132804 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:13:30.133308 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:13:30.133401 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:30.133522 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 18:13:30.133604 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:13:30.139719 10209 raft_consensus.cc:1261] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 1 FOLLOWER]: Refusing update from remote peer 6dd72cbcd08a4ad2846c4a62dc96ff35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:13:30.139710 9939 raft_consensus.cc:1261] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Refusing update from remote peer 6dd72cbcd08a4ad2846c4a62dc96ff35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:13:30.140328 10272 consensus_queue.cc:1035] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:13:30.140554 10278 consensus_queue.cc:1035] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 18:13:30.144507 10124 tablet.cc:2369] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:13:30.155727 9797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:13:30.155893 9797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:13:30.158504 9797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Sending DeleteTablet for 3 replicas of tablet f15f8e6cbe7e49bf96914cb637161265
I20260923 18:13:30.159291 10057 tablet_service.cc:1514] Processing DeleteTablet for tablet f15f8e6cbe7e49bf96914cb637161265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:30 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:48592
I20260923 18:13:30.159338 9919 tablet_service.cc:1514] Processing DeleteTablet for tablet f15f8e6cbe7e49bf96914cb637161265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:30 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:44390
I20260923 18:13:30.159730 10291 tablet_replica.cc:330] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: stopping tablet replica
I20260923 18:13:30.159773 10292 tablet_replica.cc:330] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: stopping tablet replica
I20260923 18:13:30.159930 10291 raft_consensus.cc:2229] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:13:30.159999 10189 tablet_service.cc:1514] Processing DeleteTablet for tablet f15f8e6cbe7e49bf96914cb637161265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:30 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:36860
I20260923 18:13:30.160076 10292 raft_consensus.cc:2229] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:13:30.160202 10291 raft_consensus.cc:2258] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:13:30.160408 10292 raft_consensus.cc:2258] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:13:30.160851 10293 tablet_replica.cc:330] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: stopping tablet replica
I20260923 18:13:30.160863 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 9835
I20260923 18:13:30.161152 10293 raft_consensus.cc:2229] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:13:30.161445 10293 raft_consensus.cc:2258] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:13:30.163110 10292 ts_tablet_manager.cc:1905] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:13:30.163779 10293 ts_tablet_manager.cc:1905] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:13:30.165410 10292 ts_tablet_manager.cc:1918] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:13:30.165508 10292 log.cc:1196] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/f15f8e6cbe7e49bf96914cb637161265
I20260923 18:13:30.165774 10292 ts_tablet_manager.cc:1939] T f15f8e6cbe7e49bf96914cb637161265 P 6dd72cbcd08a4ad2846c4a62dc96ff35: Deleting consensus metadata
I20260923 18:13:30.166782 10293 ts_tablet_manager.cc:1918] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:13:30.166894 10293 log.cc:1196] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/f15f8e6cbe7e49bf96914cb637161265
I20260923 18:13:30.167196 10293 ts_tablet_manager.cc:1939] T f15f8e6cbe7e49bf96914cb637161265 P f9d427254ec04ecb91796e849092517d: Deleting consensus metadata
I20260923 18:13:30.167415 9784 catalog_manager.cc:4841] TS 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811): tablet f15f8e6cbe7e49bf96914cb637161265 (table test-table [id=923987c451164931b894eb217584cf82]) successfully deleted
W20260923 18:13:30.168603 9784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:44461 (error 108)
I20260923 18:13:30.169268 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 9989
I20260923 18:13:30.169795 9784 catalog_manager.cc:4841] TS f9d427254ec04ecb91796e849092517d (127.27.75.195:32787): tablet f15f8e6cbe7e49bf96914cb637161265 (table test-table [id=923987c451164931b894eb217584cf82]) successfully deleted
I20260923 18:13:30.170761 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.169665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:44461 (local address 127.0.0.1:44396)
0923 18:13:30.170596 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:30.170673 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:44461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":843}
W20260923 18:13:30.170915 9784 catalog_manager.cc:4584] TS 1b28b788863641d884c16c6e41fc03fb (127.27.75.193:44461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f15f8e6cbe7e49bf96914cb637161265: Network error: Client connection negotiation failed: client connection to 127.27.75.193:44461: connect: Connection refused (error 111)
I20260923 18:13:30.174829 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 10126
I20260923 18:13:30.180058 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 9764
I20260923 18:13:30.185572 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39307
--webserver_interface=127.27.75.254
--webserver_port=35323
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:30.298247 10295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:30.298552 10295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:30.298595 10295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:30.302762 10295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:13:30.302839 10295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:30.302860 10295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:30.302875 10295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:13:30.302889 10295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:13:30.308535 10295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:39307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=35323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:30.309752 10295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:30.310819 10295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:30.319847 10301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:30.319847 10303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:30.319939 10295 server_base.cc:1034] running on GCE node
W20260923 18:13:30.319854 10300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:30.320946 10295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:30.322239 10295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:30.323437 10295 hybrid_clock.cc:648] HybridClock initialized: now 1790187210323410 us; error 46 us; skew 500 ppm
Sep 23 18:13:30 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187210, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:30.326431 10295 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:13:30.327437 10295 webserver.cc:460] Webserver started at http://127.27.75.254:35323/ using document root <none> and password file <none>
I20260923 18:13:30.327987 10295 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:30.328061 10295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:30.332224 10295 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:13:30.334717 10310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:30.336172 10295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:13:30.336300 10295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "38f41e55c743454a9a9d02f3805702f4"
format_stamp: "Formatted at 2026-09-23 18:13:27 on dist-test-slave-wsx5"
server_key: "592d78a1e671c48c6ebc8a8f046502e1"
server_key_iv: "649b3339777494cd204735e3bf1e3860"
server_key_version: "encryptionkey@0"
I20260923 18:13:30.336917 10295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:30.359197 10295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:30.362738 10295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:30.362908 10295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:30.370000 10295 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:39307
I20260923 18:13:30.370312 10362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:39307 every 8 connection(s)
I20260923 18:13:30.371096 10295 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:13:30.372313 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 10295
I20260923 18:13:30.372762 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:44461
--local_ip_for_outbound_sockets=127.27.75.193
--tserver_master_addrs=127.27.75.254:39307
--webserver_port=35903
--webserver_interface=127.27.75.193
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:13:30.376374 10363 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:13:30.378852 10363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Bootstrap starting.
I20260923 18:13:30.398095 10363 log.cc:824] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:30.409308 10363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:13:30.409709 10363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4: Bootstrap complete.
I20260923 18:13:30.413609 10363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:30.414016 10363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f41e55c743454a9a9d02f3805702f4, State: Initialized, Role: FOLLOWER
I20260923 18:13:30.414491 10363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:30.414615 10363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:13:30.414680 10363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:13:30.414772 10363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 2 FOLLOWER]: Advancing to term 3
I20260923 18:13:30.416706 10363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:30.417027 10363 leader_election.cc:302] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38f41e55c743454a9a9d02f3805702f4; no voters:
I20260923 18:13:30.417356 10363 leader_election.cc:288] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 18:13:30.417490 10367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 18:13:30.417694 10367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [term 3 LEADER]: Becoming Leader. State: Replica: 38f41e55c743454a9a9d02f3805702f4, State: Running, Role: LEADER
I20260923 18:13:30.417987 10367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } }
I20260923 18:13:30.418624 10363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:13:30.419329 10368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "38f41e55c743454a9a9d02f3805702f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } } }
I20260923 18:13:30.419461 10368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:30.419742 10369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f41e55c743454a9a9d02f3805702f4. Latest consensus state: current_term: 3 leader_uuid: "38f41e55c743454a9a9d02f3805702f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f41e55c743454a9a9d02f3805702f4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39307 } } }
I20260923 18:13:30.419826 10369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f41e55c743454a9a9d02f3805702f4 [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:30.420277 10374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:13:30.425341 10374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:13:30.425978 10374 catalog_manager.cc:1259] Loaded cluster ID: f49217422d81469d9e4bf41aa928b8b0
I20260923 18:13:30.426040 10374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 18:13:30.427343 10374 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f129f1a09ad google::LogMessage::Fail() at ??:0
@ 0x7f129f1a388a google::LogMessage::SendToLog() at ??:0
@ 0x7f129f1a04de google::LogMessage::Flush() at ??:0
@ 0x7f129f1a1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f12a9ae3fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f12a9ae4d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f12a9ae1579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f12a9b25640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f12a13f23d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f12a13f2c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f12a13f43f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f12a13e3512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f12a2a276db start_thread at ??:0
@ 0x7f129dc9e61f clone at ??:0
W20260923 18:13:30.518265 10365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:30.518554 10365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:30.518602 10365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:30.522835 10365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:30.522909 10365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:30.522974 10365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:13:30.528488 10365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:44461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=35903
--tserver_master_addrs=127.27.75.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:30.529680 10365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:30.530723 10365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:30.540452 10407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:30.540417 10404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:30.540726 10405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:30.540886 10365 server_base.cc:1034] running on GCE node
I20260923 18:13:30.541471 10365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:30.542403 10365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:30.543622 10365 hybrid_clock.cc:648] HybridClock initialized: now 1790187210543584 us; error 48 us; skew 500 ppm
Sep 23 18:13:30 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187210, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:30.547212 10365 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:13:30.548475 10365 webserver.cc:460] Webserver started at http://127.27.75.193:35903/ using document root <none> and password file <none>
I20260923 18:13:30.549190 10365 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:30.549264 10365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:30.554212 10365 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:13:30.556785 10414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:30.558501 10365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:13:30.558651 10365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "1b28b788863641d884c16c6e41fc03fb"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "b1adbf168f0cc5818fa287b94c98a2bc"
server_key_iv: "cfc170784cfc775ad5063fbabf6bc96c"
server_key_version: "encryptionkey@0"
I20260923 18:13:30.559252 10365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:30.578181 10365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:30.582352 10365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:30.582571 10365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:30.583328 10365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:30.584692 10421 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 18:13:30.588857 10365 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 18:13:30.588965 10365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260923 18:13:30.589051 10365 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 18:13:30.591485 10365 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 18:13:30.591557 10365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260923 18:13:30.591905 10421 tablet_bootstrap.cc:492] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: Bootstrap starting.
I20260923 18:13:30.602159 10365 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:44461
I20260923 18:13:30.602507 10528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:44461 every 8 connection(s)
I20260923 18:13:30.603293 10365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:13:30.611106 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 10365
I20260923 18:13:30.611528 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:43811
--local_ip_for_outbound_sockets=127.27.75.194
--tserver_master_addrs=127.27.75.254:39307
--webserver_port=41755
--webserver_interface=127.27.75.194
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:13:30.612708 10421 log.cc:824] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:30.621464 10421 tablet_bootstrap.cc:492] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:13:30.621833 10421 tablet_bootstrap.cc:492] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: Bootstrap complete.
I20260923 18:13:30.623157 10421 ts_tablet_manager.cc:1397] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.011s
I20260923 18:13:30.626705 10421 raft_consensus.cc:348] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:30.627287 10421 raft_consensus.cc:729] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b28b788863641d884c16c6e41fc03fb, State: Initialized, Role: FOLLOWER
I20260923 18:13:30.627892 10421 consensus_queue.cc:260] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:30.629072 10421 ts_tablet_manager.cc:1428] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260923 18:13:30.676980 10531 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:13:30.605368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:45653)
0923 18:13:30.605879 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:30.605935 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:30.606801 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:30.676008 (+ 69207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":136,"threads_started":1}
W20260923 18:13:30.677585 10529 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:39307 (0 consecutive failures): Network error: Failed to ping master at 127.27.75.254:39307: Client connection negotiation failed: client connection to 127.27.75.254:39307: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 18:13:30.678324 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.678054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:35617)
0923 18:13:30.678185 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:30.678259 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 18:13:30.678958 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.678761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:52565)
0923 18:13:30.678857 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:30.678904 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 18:13:30.679121 10529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:13:30.728025 10532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:30.728313 10532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:30.728359 10532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:30.732406 10532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:30.732470 10532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:30.732534 10532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:13:30.738060 10532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:43811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=41755
--tserver_master_addrs=127.27.75.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:30.739089 10532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:30.740144 10532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:30.749712 10539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:30.749888 10532 server_base.cc:1034] running on GCE node
W20260923 18:13:30.749714 10542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:30.749819 10540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:30.750459 10532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:30.751127 10532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:30.752297 10532 hybrid_clock.cc:648] HybridClock initialized: now 1790187210752279 us; error 33 us; skew 500 ppm
Sep 23 18:13:30 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187210, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:30.755051 10532 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:13:30.756016 10532 webserver.cc:460] Webserver started at http://127.27.75.194:41755/ using document root <none> and password file <none>
I20260923 18:13:30.756553 10532 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:30.756623 10532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:30.760558 10532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:13:30.762712 10549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:30.763972 10532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:13:30.764112 10532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "ee3d75bbfec5558f0d0146c57452f8bd"
server_key_iv: "e580a9e5fe4701bd5e74280bfa82cf91"
server_key_version: "encryptionkey@0"
I20260923 18:13:30.764684 10532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:30.785988 10532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:30.789194 10532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:30.789384 10532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:30.789969 10532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:30.790807 10532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:30.790851 10532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:30.790904 10532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:30.790932 10532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:30.800233 10532 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:43811
I20260923 18:13:30.800253 10662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:43811 every 8 connection(s)
I20260923 18:13:30.801262 10532 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:13:30.801663 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 10532
I20260923 18:13:30.802079 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:32787
--local_ip_for_outbound_sockets=127.27.75.195
--tserver_master_addrs=127.27.75.254:39307
--webserver_port=33491
--webserver_interface=127.27.75.195
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:13:30.805625 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.803420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:42313)
0923 18:13:30.803938 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:30.804098 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
W20260923 18:13:30.806140 10663 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:39307 (0 consecutive failures): Network error: Failed to ping master at 127.27.75.254:39307: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
I20260923 18:13:30.806803 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.806538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:42423)
0923 18:13:30.806672 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:30.806732 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 18:13:30.807483 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:30.807254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:35361)
0923 18:13:30.807369 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:30.807421 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 18:13:30.807693 10663 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:13:30.916886 10665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:30.917246 10665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:30.917299 10665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:30.921448 10665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:30.921525 10665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:30.921586 10665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:13:30.927104 10665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:32787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=33491
--tserver_master_addrs=127.27.75.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:30.928196 10665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:30.929239 10665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:30.939864 10671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:30.939862 10674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:30.939937 10665 server_base.cc:1034] running on GCE node
W20260923 18:13:30.939862 10672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:30.940701 10665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:30.941455 10665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:30.942673 10665 hybrid_clock.cc:648] HybridClock initialized: now 1790187210942648 us; error 40 us; skew 500 ppm
Sep 23 18:13:30 dist-test-slave-wsx5 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187210, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:30.945875 10665 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:13:30.946969 10665 webserver.cc:460] Webserver started at http://127.27.75.195:33491/ using document root <none> and password file <none>
I20260923 18:13:30.947523 10665 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:30.947587 10665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:30.951601 10665 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:13:30.953874 10681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:30.955250 10665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:13:30.955363 10665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "f9d427254ec04ecb91796e849092517d"
format_stamp: "Formatted at 2026-09-23 18:13:28 on dist-test-slave-wsx5"
server_key: "5d20f082dbf67b6e59ec439a4b0a04c5"
server_key_iv: "3176cb584435ed5126bb2334feaae99c"
server_key_version: "encryptionkey@0"
I20260923 18:13:30.956005 10665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:30.990753 10665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:30.994146 10665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:30.994326 10665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:30.994961 10665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:30.995836 10665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:30.995884 10665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:30.995926 10665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:30.995955 10665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:31.004969 10665 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:32787
I20260923 18:13:31.005003 10794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:32787 every 8 connection(s)
I20260923 18:13:31.006016 10665 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:13:31.009521 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 10665
I20260923 18:13:31.010097 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:31.008244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:45175)
0923 18:13:31.008711 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:31.009156 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
W20260923 18:13:31.010608 10795 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:39307 (0 consecutive failures): Network error: Failed to ping master at 127.27.75.254:39307: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
I20260923 18:13:31.011200 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:31.010971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:44425)
0923 18:13:31.011077 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:31.011123 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 18:13:31.011813 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:31.011626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:50297)
0923 18:13:31.011723 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:31.011764 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 18:13:31.011968 10795 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 18:13:31.680284 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:31.679806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:50353)
0923 18:13:31.680112 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:31.680180 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260923 18:13:31.808830 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:31.808349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:34991)
0923 18:13:31.808649 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:31.808716 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
I20260923 18:13:32.012972 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:32.012526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:33385)
0923 18:13:32.012794 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:32.012852 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20260923 18:13:32.429019 10802 raft_consensus.cc:482] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:13:32.429309 10802 raft_consensus.cc:504] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:32.430436 10802 leader_election.cc:288] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811), f9d427254ec04ecb91796e849092517d (127.27.75.195:32787)
Sep 23 18:13:32 dist-test-slave-wsx5 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187210, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.195@KRBTEST.COM
Sep 23 18:13:32 dist-test-slave-wsx5 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187210, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.194@KRBTEST.COM
I20260923 18:13:32.446166 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:32.431205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:32787 (local address 127.27.75.193:34503)
0923 18:13:32.431492 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:32.431505 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:32.431604 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:32.433994 (+ 2390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:32.433996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:32.434189 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:32.434598 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:32.434614 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:32.435903 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:32.435906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:32.436839 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:32.436850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:32.437611 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:32.437614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:32.437871 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:32.438550 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:32.438562 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:32.440238 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:32.442805 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:32.442813 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:32.442829 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:32.443303 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:32.443761 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:32.443765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:32.443767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:32.444011 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:32.444983 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:32.444986 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:32.445317 (+ 331us) client_negotiation.cc:769] Sending connection context
0923 18:13:32.445564 (+ 247us) client_negotiation.cc:240] Negotiation successful
0923 18:13:32.446017 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":360,"spinlock_wait_cycles":34688,"thread_start_us":63,"threads_started":1}
I20260923 18:13:32.446238 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:32.430981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:41615 (local address 127.27.75.194:43811)
0923 18:13:32.431414 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:32.431442 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:32.431489 (+ 47us) server_negotiation.cc:407] Connection header received
0923 18:13:32.432595 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:32.432604 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:32.433147 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:32.433506 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:32.434783 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:32.435749 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:32.436962 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:32.437489 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:32.437644 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:32.440731 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:32.440801 (+ 70us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:32.440824 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:32.440862 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:32.442820 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:32.443430 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:32.443437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:32.443443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:32.443632 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:32.444148 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:32.444152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:32.444153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:32.444757 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:32.444894 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:32.445726 (+ 832us) server_negotiation.cc:299] Negotiation successful
0923 18:13:32.446045 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":100,"threads_started":1}
I20260923 18:13:32.446238 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:32.431423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:34503 (local address 127.27.75.195:32787)
0923 18:13:32.431837 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:32.431859 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:32.431897 (+ 38us) server_negotiation.cc:407] Connection header received
0923 18:13:32.432785 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:32.432793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:32.433215 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:32.433506 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:32.434774 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:32.435753 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:32.436966 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:32.437488 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:32.437643 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:32.440512 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:32.440518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:32.440533 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:32.440570 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:32.442613 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:32.443430 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:32.443437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:32.443444 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:32.443623 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:32.444158 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:32.444162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:32.444163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:32.444757 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:32.444894 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:32.445726 (+ 832us) server_negotiation.cc:299] Negotiation successful
0923 18:13:32.446054 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":111,"threads_started":1}
I20260923 18:13:32.446166 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:32.430772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:43811 (local address 127.27.75.193:41615)
0923 18:13:32.431019 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:32.431033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:32.431132 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:32.433604 (+ 2472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:32.433609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:32.434091 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:32.434597 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:32.434614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:32.435905 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:32.435908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:32.436838 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:32.436849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:32.437611 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:32.437615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:32.437869 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:32.438516 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:32.438541 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:32.440512 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:32.442954 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:32.442957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:32.442960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:32.443303 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:32.443761 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:32.443765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:32.443767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:32.444011 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:32.444926 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:32.444939 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:32.445317 (+ 378us) client_negotiation.cc:769] Sending connection context
0923 18:13:32.445564 (+ 247us) client_negotiation.cc:240] Negotiation successful
0923 18:13:32.445950 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":115,"spinlock_wait_cycles":21248,"thread_start_us":66,"threads_started":1}
W20260923 18:13:32.448756 10418 leader_election.cc:341] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9d427254ec04ecb91796e849092517d (127.27.75.195:32787): Not found: Tablet not found: f15f8e6cbe7e49bf96914cb637161265
W20260923 18:13:32.448997 10418 leader_election.cc:341] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811): Not found: Tablet not found: f15f8e6cbe7e49bf96914cb637161265
I20260923 18:13:32.449088 10418 leader_election.cc:302] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b28b788863641d884c16c6e41fc03fb; no voters: 6dd72cbcd08a4ad2846c4a62dc96ff35, f9d427254ec04ecb91796e849092517d
I20260923 18:13:32.449386 10802 raft_consensus.cc:2726] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:13:32.681468 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:32.681164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:36543)
0923 18:13:32.681293 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:32.681363 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 18:13:32.810042 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:32.809625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:52965)
0923 18:13:32.809866 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:32.809919 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260923 18:13:33.014389 10808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:33.013879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:56831)
0923 18:13:33.014212 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:33.014286 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
I20260923 18:13:33.682770 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:33.682299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:37285)
0923 18:13:33.682597 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:33.682670 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
I20260923 18:13:33.811328 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:33.810853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:50899)
0923 18:13:33.811162 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:33.811228 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260923 18:13:34.015865 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:34.015350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:35021)
0923 18:13:34.015695 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:34.015766 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260923 18:13:34.078635 10812 raft_consensus.cc:482] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:13:34.078773 10812 raft_consensus.cc:504] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:34.079236 10812 leader_election.cc:288] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811), f9d427254ec04ecb91796e849092517d (127.27.75.195:32787)
W20260923 18:13:34.080143 10418 leader_election.cc:341] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811): Not found: Tablet not found: f15f8e6cbe7e49bf96914cb637161265
W20260923 18:13:34.080263 10418 leader_election.cc:341] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9d427254ec04ecb91796e849092517d (127.27.75.195:32787): Not found: Tablet not found: f15f8e6cbe7e49bf96914cb637161265
I20260923 18:13:34.080300 10418 leader_election.cc:302] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b28b788863641d884c16c6e41fc03fb; no voters: 6dd72cbcd08a4ad2846c4a62dc96ff35, f9d427254ec04ecb91796e849092517d
I20260923 18:13:34.080444 10812 raft_consensus.cc:2726] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:13:34.684141 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:34.683695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:44803)
0923 18:13:34.683972 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:34.684045 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260923 18:13:34.812664 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:34.812223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:58517)
0923 18:13:34.812471 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:34.812546 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260923 18:13:35.017194 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:35.016664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:46581)
0923 18:13:35.016975 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:35.017046 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
I20260923 18:13:35.685813 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:35.685334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:51891)
0923 18:13:35.685653 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:35.685719 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":76,"threads_started":1}
I20260923 18:13:35.813908 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:35.813499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:40309)
0923 18:13:35.813746 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:35.813807 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260923 18:13:36.018572 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:36.018101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:38227)
0923 18:13:36.018384 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:36.018474 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260923 18:13:36.163805 10819 raft_consensus.cc:482] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:13:36.163964 10819 raft_consensus.cc:504] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:36.164404 10819 leader_election.cc:288] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811), f9d427254ec04ecb91796e849092517d (127.27.75.195:32787)
W20260923 18:13:36.165426 10418 leader_election.cc:341] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9d427254ec04ecb91796e849092517d (127.27.75.195:32787): Not found: Tablet not found: f15f8e6cbe7e49bf96914cb637161265
W20260923 18:13:36.165601 10418 leader_election.cc:341] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811): Not found: Tablet not found: f15f8e6cbe7e49bf96914cb637161265
I20260923 18:13:36.165654 10418 leader_election.cc:302] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b28b788863641d884c16c6e41fc03fb; no voters: 6dd72cbcd08a4ad2846c4a62dc96ff35, f9d427254ec04ecb91796e849092517d
I20260923 18:13:36.165809 10819 raft_consensus.cc:2726] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:13:36.687073 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:36.686611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:41047)
0923 18:13:36.686877 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:36.686976 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260923 18:13:36.815246 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:36.814812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:51115)
0923 18:13:36.815077 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:36.815146 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260923 18:13:37.019826 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:37.019382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:38137)
0923 18:13:37.019670 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:37.019731 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260923 18:13:37.688326 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:37.687870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:53227)
0923 18:13:37.688167 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:37.688233 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
I20260923 18:13:37.816574 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:37.816095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:48619)
0923 18:13:37.816413 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:37.816478 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":84,"threads_started":1}
I20260923 18:13:38.021250 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:38.020716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:60831)
0923 18:13:38.021044 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:38.021107 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":76,"threads_started":1}
I20260923 18:13:38.689656 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:38.689180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:59597)
0923 18:13:38.689500 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:38.689561 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":81,"threads_started":1}
I20260923 18:13:38.817859 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:38.817390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:36967)
0923 18:13:38.817698 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:38.817761 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":84,"threads_started":1}
I20260923 18:13:39.022406 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:39.022026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:58853)
0923 18:13:39.022262 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:39.022312 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260923 18:13:39.690944 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:39.690457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:54985)
0923 18:13:39.690785 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:39.690848 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":99,"threads_started":1}
I20260923 18:13:39.819093 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:39.818670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:45671)
0923 18:13:39.818937 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:39.818991 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260923 18:13:40.023792 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:40.023299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:52835)
0923 18:13:40.023622 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:40.023690 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
I20260923 18:13:40.542575 10832 raft_consensus.cc:482] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:13:40.542708 10832 raft_consensus.cc:504] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:40.543099 10832 leader_election.cc:288] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811), f9d427254ec04ecb91796e849092517d (127.27.75.195:32787)
W20260923 18:13:40.543926 10418 leader_election.cc:341] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811): Not found: Tablet not found: f15f8e6cbe7e49bf96914cb637161265
W20260923 18:13:40.544091 10418 leader_election.cc:341] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9d427254ec04ecb91796e849092517d (127.27.75.195:32787): Not found: Tablet not found: f15f8e6cbe7e49bf96914cb637161265
I20260923 18:13:40.544144 10418 leader_election.cc:302] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b28b788863641d884c16c6e41fc03fb; no voters: 6dd72cbcd08a4ad2846c4a62dc96ff35, f9d427254ec04ecb91796e849092517d
I20260923 18:13:40.544318 10832 raft_consensus.cc:2726] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:13:40.692371 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:40.691903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:35877)
0923 18:13:40.692206 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:40.692272 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
I20260923 18:13:40.820458 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:40.819949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:51985)
0923 18:13:40.820264 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:40.820333 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
I20260923 18:13:41.025611 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:41.025095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:36405)
0923 18:13:41.025422 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:41.025492 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260923 18:13:41.693760 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:41.693277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:59841)
0923 18:13:41.693589 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:41.693653 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":84,"threads_started":1}
I20260923 18:13:41.821709 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:41.821249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:54215)
0923 18:13:41.821558 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:41.821610 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":152,"threads_started":1}
I20260923 18:13:42.026865 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:42.026422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:44077)
0923 18:13:42.026700 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:42.026764 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260923 18:13:42.695087 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:42.694577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:36835)
0923 18:13:42.694871 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:42.694991 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
I20260923 18:13:42.822980 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:42.822507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:51715)
0923 18:13:42.822813 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:42.822875 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":115,"threads_started":1}
I20260923 18:13:42.946496 10841 raft_consensus.cc:482] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:13:42.946647 10841 raft_consensus.cc:504] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd72cbcd08a4ad2846c4a62dc96ff35" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43811 } } peers { permanent_uuid: "f9d427254ec04ecb91796e849092517d" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 32787 } } peers { permanent_uuid: "1b28b788863641d884c16c6e41fc03fb" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 44461 } }
I20260923 18:13:42.947057 10841 leader_election.cc:288] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811), f9d427254ec04ecb91796e849092517d (127.27.75.195:32787)
W20260923 18:13:42.947896 10418 leader_election.cc:341] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6dd72cbcd08a4ad2846c4a62dc96ff35 (127.27.75.194:43811): Not found: Tablet not found: f15f8e6cbe7e49bf96914cb637161265
W20260923 18:13:42.948014 10418 leader_election.cc:341] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9d427254ec04ecb91796e849092517d (127.27.75.195:32787): Not found: Tablet not found: f15f8e6cbe7e49bf96914cb637161265
I20260923 18:13:42.948067 10418 leader_election.cc:302] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b28b788863641d884c16c6e41fc03fb; no voters: 6dd72cbcd08a4ad2846c4a62dc96ff35, f9d427254ec04ecb91796e849092517d
I20260923 18:13:42.948217 10841 raft_consensus.cc:2726] T f15f8e6cbe7e49bf96914cb637161265 P 1b28b788863641d884c16c6e41fc03fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:13:43.028192 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:43.027720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:41459)
0923 18:13:43.028001 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:43.028083 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260923 18:13:43.696379 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:43.695865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:51951)
0923 18:13:43.696206 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:43.696274 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":75,"threads_started":1}
I20260923 18:13:43.824133 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:43.823731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:60873)
0923 18:13:43.823978 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:43.824037 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
I20260923 18:13:44.029534 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:44.029014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:59575)
0923 18:13:44.029357 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:44.029424 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":121,"threads_started":1}
I20260923 18:13:44.697817 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:44.697384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:50335)
0923 18:13:44.697651 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:44.697717 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260923 18:13:44.825476 10847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:44.824984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:60517)
0923 18:13:44.825307 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:44.825373 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":84,"threads_started":1}
I20260923 18:13:45.030791 10848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:45.030367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.195:50443)
0923 18:13:45.030631 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:45.030687 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260923 18:13:45.699378 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:45.698838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.193:42711)
0923 18:13:45.699184 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:45.699272 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
I20260923 18:13:45.826818 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:45.826383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39307 (local address 127.27.75.194:41105)
0923 18:13:45.826636 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:45.826697 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
I20260923 18:13:46.014140 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 10365
I20260923 18:13:46.019656 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 10532
I20260923 18:13:46.024924 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 10665
2026-09-23T18:13:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19451 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:13:46 dist-test-slave-wsx5 krb5kdc[10853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:13:46 dist-test-slave-wsx5 krb5kdc[10853](info): set up 2 sockets
Sep 23 18:13:46 dist-test-slave-wsx5 krb5kdc[10853](info): commencing operation
krb5kdc: starting...
W20260923 18:13:46.991515 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:13:47 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:13:47.116596 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42163
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:42163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:47.229876 10866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:47.230333 10866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:47.230470 10866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:47.234858 10866 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 18:13:47.234937 10866 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 18:13:47.234958 10866 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 18:13:47.234977 10866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:13:47.235002 10866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:47.235028 10866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:47.235059 10866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:13:47.235090 10866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:13:47.240723 10866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:42163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:47.242046 10866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:47.243183 10866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:47.252312 10874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:47.252270 10871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:47.252345 10872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:47.252504 10866 server_base.cc:1034] running on GCE node
I20260923 18:13:47.253173 10866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:47.254346 10866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:47.255510 10866 hybrid_clock.cc:648] HybridClock initialized: now 1790187227255493 us; error 32 us; skew 500 ppm
Sep 23 18:13:47 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:47.258385 10866 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:13:47.259289 10866 webserver.cc:460] Webserver started at http://127.27.75.254:46177/ using document root <none> and password file <none>
I20260923 18:13:47.259846 10866 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:47.259907 10866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:47.260092 10866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:47.262293 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "6ed3020abbec41848b2ce1096718043a"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "d9206c22aa4a65b8d459d479483ffbbc"
server_key_iv: "16ef05c4e1b6ffeee2701b3f4ab4d56a"
server_key_version: "encryptionkey@0"
I20260923 18:13:47.262815 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "6ed3020abbec41848b2ce1096718043a"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "d9206c22aa4a65b8d459d479483ffbbc"
server_key_iv: "16ef05c4e1b6ffeee2701b3f4ab4d56a"
server_key_version: "encryptionkey@0"
I20260923 18:13:47.266614 10866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:13:47.269344 10881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:47.270694 10866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:13:47.270823 10866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "6ed3020abbec41848b2ce1096718043a"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "d9206c22aa4a65b8d459d479483ffbbc"
server_key_iv: "16ef05c4e1b6ffeee2701b3f4ab4d56a"
server_key_version: "encryptionkey@0"
I20260923 18:13:47.270951 10866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:47.298632 10866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:47.302029 10866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:47.302215 10866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:47.309051 10866 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:42163
I20260923 18:13:47.309358 10933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:42163 every 8 connection(s)
I20260923 18:13:47.310083 10866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:13:47.313072 10934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:47.313470 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 10866
I20260923 18:13:47.313596 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:13:47.313886 27951 external_mini_cluster.cc:1428] Setting key f30a460880604f92fe73fe536215d196
I20260923 18:13:47.321209 10934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Bootstrap starting.
Sep 23 18:13:47 dist-test-slave-wsx5 krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:47.324057 10934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:47.324950 10934 log.cc:824] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:47.327423 10934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: No bootstrap required, opened a new log
I20260923 18:13:47.329613 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:47.315295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.27.75.254:42163)
0923 18:13:47.315749 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:47.315768 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:47.315816 (+ 48us) server_negotiation.cc:407] Connection header received
0923 18:13:47.316558 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:47.316584 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:47.317042 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:47.317485 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:47.318219 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:47.319847 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:47.320837 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:47.321628 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:47.321893 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:47.324413 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:47.324420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:47.324438 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:47.324476 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:47.326272 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:47.326732 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:47.326737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:47.326743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:47.326891 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:47.327251 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:47.327254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:47.327256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:47.327790 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:47.327972 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:47.328280 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 18:13:47.328603 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":123,"threads_started":1}
I20260923 18:13:47.331894 10934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:13:47.332177 10934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:47.332216 10934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed3020abbec41848b2ce1096718043a, State: Initialized, Role: FOLLOWER
I20260923 18:13:47.332597 10934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:13:47.332723 10934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:13:47.332763 10934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:13:47.332834 10934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:47.333773 10934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:13:47.334079 10934 leader_election.cc:302] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ed3020abbec41848b2ce1096718043a; no voters:
I20260923 18:13:47.334338 10934 leader_election.cc:288] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:13:47.334486 10939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:13:47.334720 10939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 1 LEADER]: Becoming Leader. State: Replica: 6ed3020abbec41848b2ce1096718043a, State: Running, Role: LEADER
I20260923 18:13:47.335016 10939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:13:47.335508 10934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:13:47.336541 10941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ed3020abbec41848b2ce1096718043a. Latest consensus state: current_term: 1 leader_uuid: "6ed3020abbec41848b2ce1096718043a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } } }
I20260923 18:13:47.336784 10941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:47.337304 10949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:13:47.337419 10940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ed3020abbec41848b2ce1096718043a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } } }
I20260923 18:13:47.337774 10940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: This master's current role is: LEADER
I20260923 18:13:47.340057 10949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:13:47.345242 10949 catalog_manager.cc:1347] Generated new cluster ID: a5ca4600304649c3ad3a084a8ce7361b
I20260923 18:13:47.345317 10949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:13:47.365909 10949 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:13:47.367264 10949 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:13:47.380622 10949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Generated new TSK 0
I20260923 18:13:47.381606 10949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:13:47.420408 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42163
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:47.534674 10960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:47.534964 10960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:47.535010 10960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:47.539373 10960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:47.539453 10960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:47.539515 10960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:13:47.545022 10960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:47.546170 10960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:47.547194 10960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:47.557276 10965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:47.557464 10960 server_base.cc:1034] running on GCE node
W20260923 18:13:47.557276 10968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:47.557686 10966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:47.558123 10960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:47.558832 10960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:47.560104 10960 hybrid_clock.cc:648] HybridClock initialized: now 1790187227560073 us; error 43 us; skew 500 ppm
Sep 23 18:13:47 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:47.563032 10960 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:13:47.564050 10960 webserver.cc:460] Webserver started at http://127.27.75.193:35747/ using document root <none> and password file <none>
I20260923 18:13:47.564639 10960 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:47.564708 10960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:47.564908 10960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:47.567299 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "c163e090e5ac4b45a98da3fc9f72baec"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "934b34b6ce2fd570119870b2ede8584f"
server_key_iv: "de7f0c28f0108a696fefe61e97795b61"
server_key_version: "encryptionkey@0"
I20260923 18:13:47.567937 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "c163e090e5ac4b45a98da3fc9f72baec"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "934b34b6ce2fd570119870b2ede8584f"
server_key_iv: "de7f0c28f0108a696fefe61e97795b61"
server_key_version: "encryptionkey@0"
I20260923 18:13:47.572191 10960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 18:13:47.575121 10975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:47.576640 10960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:13:47.576777 10960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "c163e090e5ac4b45a98da3fc9f72baec"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "934b34b6ce2fd570119870b2ede8584f"
server_key_iv: "de7f0c28f0108a696fefe61e97795b61"
server_key_version: "encryptionkey@0"
I20260923 18:13:47.576927 10960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:47.593026 10960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:47.596493 10960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:47.596670 10960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:47.597328 10960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:47.598201 10960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:47.598251 10960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:47.598292 10960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:47.598307 10960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:47.606838 10960 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:33403
I20260923 18:13:47.606859 11089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:33403 every 8 connection(s)
I20260923 18:13:47.607779 10960 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
Sep 23 18:13:47 dist-test-slave-wsx5 krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:47.617275 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 10960
I20260923 18:13:47.617390 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:13:47.617659 27951 external_mini_cluster.cc:1428] Setting key b9611e9ce405ff5a3bb25a98c7c27265
I20260923 18:13:47.621762 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:47.609688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:49675 (local address 127.27.75.254:42163)
0923 18:13:47.609857 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:47.609862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:47.610722 (+ 860us) server_negotiation.cc:407] Connection header received
0923 18:13:47.612005 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:47.612008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:47.612058 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:47.612137 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:47.613435 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:47.613985 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:47.615114 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:47.615338 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:47.615413 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:47.617903 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:47.617907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:47.617910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:47.617941 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:47.619494 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:47.620004 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:47.620008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:47.620009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:47.620111 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:47.620412 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:47.620415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:47.620416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:47.620626 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:47.620729 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:47.621494 (+ 765us) server_negotiation.cc:299] Negotiation successful
0923 18:13:47.621635 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 18:13:47.622826 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:47.610099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:49675)
0923 18:13:47.610560 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:47.610606 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:47.611761 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:47.612300 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:47.612306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:47.612667 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:47.613267 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:47.613284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:47.614109 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:47.614112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:47.615010 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:47.615020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:47.615456 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:47.615458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:47.615606 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:47.616198 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:47.616215 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:47.617692 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:47.619613 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:47.619618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:47.619633 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:47.619907 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:47.620193 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:47.620195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:47.620197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:47.620337 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:47.620770 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:47.620778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:47.621174 (+ 396us) client_negotiation.cc:769] Sending connection context
0923 18:13:47.621411 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 18:13:47.621830 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260923 18:13:47.624166 11090 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42163
I20260923 18:13:47.624411 11090 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:47.624987 11090 heartbeater.cc:511] Master 127.27.75.254:42163 requested a full tablet report, sending...
I20260923 18:13:47.626881 10898 ts_manager.cc:194] Registered new tserver with Master: c163e090e5ac4b45a98da3fc9f72baec (127.27.75.193:33403)
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:13:47.628276 10898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:49675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:13:47.646384 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42163
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:47.759778 11095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:47.760089 11095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:47.760135 11095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:47.764433 11095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:47.764524 11095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:47.764591 11095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:13:47.770566 11095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:47.772745 11095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:47.774443 11095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:13:47.785759 11095 server_base.cc:1034] running on GCE node
W20260923 18:13:47.785697 11100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:47.785710 11103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:47.785697 11101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:47.786458 11095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:47.787158 11095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:47.788352 11095 hybrid_clock.cc:648] HybridClock initialized: now 1790187227788338 us; error 28 us; skew 500 ppm
Sep 23 18:13:47 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:47.791393 11095 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:13:47.792449 11095 webserver.cc:460] Webserver started at http://127.27.75.194:35449/ using document root <none> and password file <none>
I20260923 18:13:47.793048 11095 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:47.793146 11095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:47.793354 11095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:47.795498 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "6f433495bcd74994bf41c2b7ed29d3de"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "8b019351162935cbae572ce7541ffb11"
server_key_iv: "551427aa5ba6c6fff6fee04341f8f382"
server_key_version: "encryptionkey@0"
I20260923 18:13:47.796046 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "6f433495bcd74994bf41c2b7ed29d3de"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "8b019351162935cbae572ce7541ffb11"
server_key_iv: "551427aa5ba6c6fff6fee04341f8f382"
server_key_version: "encryptionkey@0"
I20260923 18:13:47.799850 11095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260923 18:13:47.802485 11110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:47.804034 11095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:13:47.804152 11095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "6f433495bcd74994bf41c2b7ed29d3de"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "8b019351162935cbae572ce7541ffb11"
server_key_iv: "551427aa5ba6c6fff6fee04341f8f382"
server_key_version: "encryptionkey@0"
I20260923 18:13:47.804359 11095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:47.819087 11095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:47.822373 11095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:47.822546 11095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:47.823184 11095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:47.824036 11095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:47.824083 11095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:47.824123 11095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:47.824138 11095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:47.832659 11095 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:41813
I20260923 18:13:47.832751 11225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:41813 every 8 connection(s)
I20260923 18:13:47.833593 11095 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:13:47.843346 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 11095
I20260923 18:13:47.843478 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:13:47.843803 27951 external_mini_cluster.cc:1428] Setting key a12bb97b3c031fe1847d06cd7e35d13b
Sep 23 18:13:47 dist-test-slave-wsx5 krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:47.850889 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:47.835484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:40935 (local address 127.27.75.254:42163)
0923 18:13:47.835606 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:47.835610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:47.836671 (+ 1061us) server_negotiation.cc:407] Connection header received
0923 18:13:47.838077 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:47.838080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:47.838140 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:47.838219 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:47.839461 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:47.840042 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:47.841413 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:47.841679 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:47.841776 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:47.846154 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:47.846160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:47.846163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:47.846232 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:47.848334 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:47.848997 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:47.849000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:47.849002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:47.849102 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:47.849590 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:47.849593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:47.849594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:47.849812 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:47.849913 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:47.850584 (+ 671us) server_negotiation.cc:299] Negotiation successful
0923 18:13:47.850722 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 18:13:47.852025 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:47.835930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:40935)
0923 18:13:47.836488 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:47.836561 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:47.837835 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:47.838375 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:47.838383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:47.838777 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:47.839273 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:47.839286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:47.840198 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:47.840202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:47.841264 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:47.841276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:47.841828 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:47.841831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:47.841991 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:47.842754 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:47.842783 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:47.845881 (+ 3098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:47.848475 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:47.848481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:47.848495 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:47.848876 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:47.849255 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:47.849258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:47.849261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:47.849457 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:47.849925 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:47.849932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:47.850293 (+ 361us) client_negotiation.cc:769] Sending connection context
0923 18:13:47.850515 (+ 222us) client_negotiation.cc:240] Negotiation successful
0923 18:13:47.850887 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":128,"threads_started":1}
I20260923 18:13:47.853298 11226 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42163
I20260923 18:13:47.853592 11226 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:47.854235 11226 heartbeater.cc:511] Master 127.27.75.254:42163 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:13:47.855460 10898 ts_manager.cc:194] Registered new tserver with Master: 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813)
I20260923 18:13:47.856072 10898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:40935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:13:47.876679 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42163
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:13:47.990388 11231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:13:47.990694 11231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:13:47.990739 11231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:13:47.995044 11231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:13:47.995123 11231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:13:47.995186 11231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:13:48.000787 11231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:13:48.001958 11231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:13:48.003005 11231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:13:48.013258 11237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:48.013288 11236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:13:48.013280 11239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:13:48.013623 11231 server_base.cc:1034] running on GCE node
I20260923 18:13:48.014091 11231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:13:48.014777 11231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:13:48.015960 11231 hybrid_clock.cc:648] HybridClock initialized: now 1790187228015941 us; error 35 us; skew 500 ppm
Sep 23 18:13:48 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187228, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:13:48.018862 11231 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:13:48.019867 11231 webserver.cc:460] Webserver started at http://127.27.75.195:34047/ using document root <none> and password file <none>
I20260923 18:13:48.020450 11231 fs_manager.cc:359] Metadata directory not provided
I20260923 18:13:48.020521 11231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:13:48.020743 11231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:13:48.022909 11231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "3b12bf3040754caf8e9269458ea1aa32"
format_stamp: "Formatted at 2026-09-23 18:13:48 on dist-test-slave-wsx5"
server_key: "71841c4e3aa6477ee55a2ce255216467"
server_key_iv: "ba3991cf09c342686774c63c2c0293ff"
server_key_version: "encryptionkey@0"
I20260923 18:13:48.023469 11231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "3b12bf3040754caf8e9269458ea1aa32"
format_stamp: "Formatted at 2026-09-23 18:13:48 on dist-test-slave-wsx5"
server_key: "71841c4e3aa6477ee55a2ce255216467"
server_key_iv: "ba3991cf09c342686774c63c2c0293ff"
server_key_version: "encryptionkey@0"
I20260923 18:13:48.027479 11231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260923 18:13:48.030071 11246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:48.031312 11231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:13:48.031437 11231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "3b12bf3040754caf8e9269458ea1aa32"
format_stamp: "Formatted at 2026-09-23 18:13:48 on dist-test-slave-wsx5"
server_key: "71841c4e3aa6477ee55a2ce255216467"
server_key_iv: "ba3991cf09c342686774c63c2c0293ff"
server_key_version: "encryptionkey@0"
I20260923 18:13:48.031560 11231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:13:48.051164 11231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:13:48.054469 11231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:13:48.054669 11231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:13:48.055331 11231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:13:48.056151 11231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:13:48.056211 11231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:48.056262 11231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:13:48.056282 11231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:13:48.065073 11231 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:41005
I20260923 18:13:48.065100 11360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:41005 every 8 connection(s)
I20260923 18:13:48.066082 11231 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:13:48.073694 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 11231
I20260923 18:13:48.073819 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:13:48.074095 27951 external_mini_cluster.cc:1428] Setting key 5bae3664108c6d54cf7006c87f0b4e4d
Sep 23 18:13:48 dist-test-slave-wsx5 krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187228, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:13:48.080646 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.068028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:59753 (local address 127.27.75.254:42163)
0923 18:13:48.068177 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:48.068181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:48.068965 (+ 784us) server_negotiation.cc:407] Connection header received
0923 18:13:48.070460 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:48.070463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:48.070532 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:48.070605 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:48.071822 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.072354 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.073632 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.073906 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.074093 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.076986 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:48.076990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:48.076992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:48.077022 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:48.078440 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:48.079015 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:48.079017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:48.079018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:48.079104 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:48.079481 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:48.079483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:48.079484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:48.079666 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:48.079739 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:48.080363 (+ 624us) server_negotiation.cc:299] Negotiation successful
0923 18:13:48.080516 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 18:13:48.081637 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.068367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:59753)
0923 18:13:48.068817 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:48.068861 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:48.070225 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:48.070792 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:48.070799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:48.071178 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:13:48.071673 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.071687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.072475 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.072478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.073471 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.073483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.074150 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.074154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.074345 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.075051 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:13:48.075075 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:13:48.076798 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:13:48.078579 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:48.078585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:48.078599 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:48.078876 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:48.079234 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:13:48.079237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:13:48.079239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:13:48.079397 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:13:48.079766 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:13:48.079778 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:13:48.080101 (+ 323us) client_negotiation.cc:769] Sending connection context
0923 18:13:48.080284 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 18:13:48.080577 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260923 18:13:48.082901 11361 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42163
I20260923 18:13:48.083155 11361 heartbeater.cc:464] Registering TS with master...
I20260923 18:13:48.083688 11361 heartbeater.cc:511] Master 127.27.75.254:42163 requested a full tablet report, sending...
I20260923 18:13:48.084851 10898 ts_manager.cc:194] Registered new tserver with Master: 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005)
I20260923 18:13:48.085515 10898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:59753
I20260923 18:13:48.088481 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:13:48.097679 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.090450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.27.75.254:42163)
0923 18:13:48.090620 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:48.090625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:48.090753 (+ 128us) server_negotiation.cc:407] Connection header received
0923 18:13:48.090897 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:48.090901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:48.090949 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:48.091061 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:13:48.091801 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.092329 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.093186 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.093421 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.093511 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.094795 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:13:48.094799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:13:48.094801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:13:48.094828 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:13:48.096058 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:48.096452 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:48.096456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:48.096457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:48.096540 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:13:48.096782 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:13:48.096786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:13:48.096788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:13:48.097026 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:13:48.097187 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:48.097401 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 18:13:48.097518 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 18:13:48.100668 10898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:13:48.103829 10898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:13:48.119136 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.113680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.27.75.193:33403)
0923 18:13:48.114050 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:48.114064 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:48.114483 (+ 419us) server_negotiation.cc:407] Connection header received
0923 18:13:48.114568 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:48.114574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:48.114794 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:48.114943 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:48.115321 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.116098 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.117786 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.118434 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.118518 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.118654 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:48.118870 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 18:13:48.118973 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":79,"threads_started":1}
I20260923 18:13:48.119138 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.113588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:41813 (local address 127.0.0.1:44684)
0923 18:13:48.114177 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:48.114191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:48.114327 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:48.114943 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:48.114949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:48.114970 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:48.115110 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.115117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.116554 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.116557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.117832 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.117840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.118639 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.118642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.118741 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.118759 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 18:13:48.118828 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 18:13:48.118979 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":101,"spinlock_wait_cycles":2304,"thread_start_us":104,"threads_started":1}
I20260923 18:13:48.119146 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.113283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:33403 (local address 127.0.0.1:34318)
0923 18:13:48.114131 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:48.114185 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:48.114327 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:48.115110 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:48.115113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:48.115128 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:48.115209 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.115214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.116226 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.116229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.117655 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.117666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.118553 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.118557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.118676 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.118715 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 18:13:48.118827 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 18:13:48.118972 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":52,"spinlock_wait_cycles":11392,"thread_start_us":87,"threads_started":1}
I20260923 18:13:48.119751 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.114101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.27.75.194:41813)
0923 18:13:48.114484 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:48.114496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:48.114516 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:13:48.114591 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:48.114596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:48.114785 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:48.114942 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:48.115231 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.116419 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.117948 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.118507 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.118619 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.118758 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:48.119519 (+ 761us) server_negotiation.cc:299] Negotiation successful
0923 18:13:48.119619 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":105,"threads_started":1}
I20260923 18:13:48.121673 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.113283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:41005 (local address 127.0.0.1:56426)
0923 18:13:48.114554 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:48.114565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:48.114634 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:48.115241 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:48.115244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:48.115259 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:48.115327 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.115332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.116687 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.116691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.117827 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.117837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.119751 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.119755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.121344 (+ 1589us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.121368 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:13:48.121448 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 18:13:48.121536 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"mutex_wait_us":469,"thread_start_us":82,"threads_started":1}
I20260923 18:13:48.121737 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.113424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56426 (local address 127.27.75.195:41005)
0923 18:13:48.114855 (+ 1431us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:48.114867 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:48.114888 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:13:48.114955 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:48.114960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:48.115119 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:48.115239 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:48.115455 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.116563 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.118828 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.119635 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.119740 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.119877 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:48.121499 (+ 1622us) server_negotiation.cc:299] Negotiation successful
0923 18:13:48.121606 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1331,"thread_start_us":145,"threads_started":1}
I20260923 18:13:48.122617 11159 tablet_service.cc:1467] Processing CreateTablet for tablet 9a822a90877a49b9ac21395d5cbe2240 (DEFAULT_TABLE table=test-table [id=67660519136747d799215b85215e3b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:48.122617 11023 tablet_service.cc:1467] Processing CreateTablet for tablet 9a822a90877a49b9ac21395d5cbe2240 (DEFAULT_TABLE table=test-table [id=67660519136747d799215b85215e3b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:48.123778 11159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a822a90877a49b9ac21395d5cbe2240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:48.123778 11023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a822a90877a49b9ac21395d5cbe2240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:48.124084 11294 tablet_service.cc:1467] Processing CreateTablet for tablet 9a822a90877a49b9ac21395d5cbe2240 (DEFAULT_TABLE table=test-table [id=67660519136747d799215b85215e3b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:13:48.125368 11294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a822a90877a49b9ac21395d5cbe2240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:13:48.131369 11378 tablet_bootstrap.cc:492] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Bootstrap starting.
I20260923 18:13:48.132319 11379 tablet_bootstrap.cc:492] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de: Bootstrap starting.
I20260923 18:13:48.133460 11380 tablet_bootstrap.cc:492] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: Bootstrap starting.
I20260923 18:13:48.135385 11379 tablet_bootstrap.cc:654] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:48.135430 11378 tablet_bootstrap.cc:654] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:48.135610 11380 tablet_bootstrap.cc:654] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: Neither blocks nor log segments found. Creating new log.
I20260923 18:13:48.136395 11380 log.cc:824] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:48.136396 11378 log.cc:824] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:48.136723 11379 log.cc:824] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de: Log is configured to *not* fsync() on all Append() calls
I20260923 18:13:48.138350 11378 tablet_bootstrap.cc:492] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: No bootstrap required, opened a new log
I20260923 18:13:48.138360 11380 tablet_bootstrap.cc:492] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: No bootstrap required, opened a new log
I20260923 18:13:48.138533 11380 ts_tablet_manager.cc:1397] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 18:13:48.138536 11378 ts_tablet_manager.cc:1397] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 18:13:48.140136 11379 tablet_bootstrap.cc:492] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de: No bootstrap required, opened a new log
I20260923 18:13:48.140329 11379 ts_tablet_manager.cc:1397] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260923 18:13:48.141961 11378 raft_consensus.cc:348] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } }
I20260923 18:13:48.142204 11378 raft_consensus.cc:374] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:48.142256 11378 raft_consensus.cc:729] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c163e090e5ac4b45a98da3fc9f72baec, State: Initialized, Role: FOLLOWER
I20260923 18:13:48.142290 11380 raft_consensus.cc:348] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } }
I20260923 18:13:48.142547 11380 raft_consensus.cc:374] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:48.142601 11380 raft_consensus.cc:729] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b12bf3040754caf8e9269458ea1aa32, State: Initialized, Role: FOLLOWER
I20260923 18:13:48.142709 11378 consensus_queue.cc:260] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } }
I20260923 18:13:48.143018 11380 consensus_queue.cc:260] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } }
I20260923 18:13:48.143918 11378 ts_tablet_manager.cc:1428] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 18:13:48.143947 11380 ts_tablet_manager.cc:1428] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 18:13:48.144280 11090 heartbeater.cc:503] Master 127.27.75.254:42163 was elected leader, sending a full tablet report...
I20260923 18:13:48.144284 11361 heartbeater.cc:503] Master 127.27.75.254:42163 was elected leader, sending a full tablet report...
I20260923 18:13:48.144547 11379 raft_consensus.cc:348] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } }
I20260923 18:13:48.144814 11379 raft_consensus.cc:374] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:13:48.144866 11379 raft_consensus.cc:729] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f433495bcd74994bf41c2b7ed29d3de, State: Initialized, Role: FOLLOWER
I20260923 18:13:48.145318 11379 consensus_queue.cc:260] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } }
I20260923 18:13:48.147372 11226 heartbeater.cc:503] Master 127.27.75.254:42163 was elected leader, sending a full tablet report...
I20260923 18:13:48.147763 11379 ts_tablet_manager.cc:1428] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 18:13:48.273367 11384 raft_consensus.cc:482] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:13:48.273595 11384 raft_consensus.cc:504] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } }
I20260923 18:13:48.274860 11384 leader_election.cc:288] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005), 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813)
I20260923 18:13:48.278842 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.275233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:47655 (local address 127.27.75.195:41005)
0923 18:13:48.275367 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:48.275372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:48.275645 (+ 273us) server_negotiation.cc:407] Connection header received
0923 18:13:48.275778 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:48.275781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:48.275832 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:48.275900 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:48.276173 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.276677 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.277872 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.278274 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.278384 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.278438 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:48.278648 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 18:13:48.278714 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:13:48.278842 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.275334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:51913 (local address 127.27.75.194:41813)
0923 18:13:48.275458 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:48.275463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:48.275757 (+ 294us) server_negotiation.cc:407] Connection header received
0923 18:13:48.275915 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:48.275917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:48.275959 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:48.276023 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:13:48.276272 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.276769 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.277876 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.278274 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.278383 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.278438 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:48.278641 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 18:13:48.278708 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:13:48.278873 11387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.275216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:41813 (local address 127.27.75.193:51913)
0923 18:13:48.275670 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:48.275682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:48.275786 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:48.276038 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:48.276041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:48.276056 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:48.276176 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.276181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.276873 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.276877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.277757 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.277766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.278383 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.278387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.278520 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.278553 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 18:13:48.278617 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 18:13:48.278698 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":47,"thread_start_us":101,"threads_started":1}
I20260923 18:13:48.278879 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.275137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:41005 (local address 127.27.75.193:47655)
0923 18:13:48.275547 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:48.275560 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:13:48.275674 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:13:48.275914 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:13:48.275917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:13:48.275932 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:13:48.276057 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.276062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.276792 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.276796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.277746 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:13:48.277757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.278385 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.278389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:13:48.278521 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.278551 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 18:13:48.278615 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 18:13:48.278700 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":41,"thread_start_us":146,"threads_started":1}
I20260923 18:13:48.279445 11179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a822a90877a49b9ac21395d5cbe2240" candidate_uuid: "c163e090e5ac4b45a98da3fc9f72baec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f433495bcd74994bf41c2b7ed29d3de" is_pre_election: true
I20260923 18:13:48.279665 11314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a822a90877a49b9ac21395d5cbe2240" candidate_uuid: "c163e090e5ac4b45a98da3fc9f72baec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b12bf3040754caf8e9269458ea1aa32" is_pre_election: true
I20260923 18:13:48.279794 11179 raft_consensus.cc:2454] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c163e090e5ac4b45a98da3fc9f72baec in term 0.
I20260923 18:13:48.279922 11314 raft_consensus.cc:2454] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c163e090e5ac4b45a98da3fc9f72baec in term 0.
I20260923 18:13:48.280344 10977 leader_election.cc:302] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f433495bcd74994bf41c2b7ed29d3de, c163e090e5ac4b45a98da3fc9f72baec; no voters:
I20260923 18:13:48.280649 11384 raft_consensus.cc:2781] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:13:48.280730 11384 raft_consensus.cc:482] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:13:48.280761 11384 raft_consensus.cc:3037] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:48.281908 11384 raft_consensus.cc:504] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } }
I20260923 18:13:48.282265 11384 leader_election.cc:288] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 1 election: Requested vote from peers 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005), 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813)
I20260923 18:13:48.282636 11314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a822a90877a49b9ac21395d5cbe2240" candidate_uuid: "c163e090e5ac4b45a98da3fc9f72baec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b12bf3040754caf8e9269458ea1aa32"
I20260923 18:13:48.282756 11314 raft_consensus.cc:3037] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:48.282757 11179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a822a90877a49b9ac21395d5cbe2240" candidate_uuid: "c163e090e5ac4b45a98da3fc9f72baec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f433495bcd74994bf41c2b7ed29d3de"
I20260923 18:13:48.282871 11179 raft_consensus.cc:3037] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:13:48.283896 11314 raft_consensus.cc:2454] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c163e090e5ac4b45a98da3fc9f72baec in term 1.
I20260923 18:13:48.284097 11179 raft_consensus.cc:2454] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c163e090e5ac4b45a98da3fc9f72baec in term 1.
I20260923 18:13:48.284250 10979 leader_election.cc:302] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b12bf3040754caf8e9269458ea1aa32, c163e090e5ac4b45a98da3fc9f72baec; no voters:
I20260923 18:13:48.284432 11384 raft_consensus.cc:2781] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:13:48.284744 11384 raft_consensus.cc:686] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 1 LEADER]: Becoming Leader. State: Replica: c163e090e5ac4b45a98da3fc9f72baec, State: Running, Role: LEADER
I20260923 18:13:48.285305 11384 consensus_queue.cc:237] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } }
I20260923 18:13:48.289016 10895 catalog_manager.cc:5495] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec reported cstate change: term changed from 0 to 1, leader changed from <none> to c163e090e5ac4b45a98da3fc9f72baec (127.27.75.193). New cstate: current_term: 1 leader_uuid: "c163e090e5ac4b45a98da3fc9f72baec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } health_report { overall_health: UNKNOWN } } }
I20260923 18:13:48.311132 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.307540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.27.75.193:33403)
0923 18:13:48.307669 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 18:13:48.307674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:13:48.307689 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:13:48.307814 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:13:48.307817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:13:48.307875 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:13:48.307980 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:13:48.308209 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.308801 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.309737 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:13:48.309963 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:13:48.310051 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:13:48.310343 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:13:48.310752 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:13:48.310858 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 18:13:48.310929 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 18:13:48.310998 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 18:13:48.317025 11362 tablet.cc:2369] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:13:48.317984 11314 raft_consensus.cc:1261] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32 [term 1 FOLLOWER]: Refusing update from remote peer c163e090e5ac4b45a98da3fc9f72baec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:13:48.317986 11179 raft_consensus.cc:1261] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de [term 1 FOLLOWER]: Refusing update from remote peer c163e090e5ac4b45a98da3fc9f72baec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:13:48.318746 11384 consensus_queue.cc:1035] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:13:48.318959 11389 consensus_queue.cc:1035] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:13:48.328964 11393 mvcc.cc:205] Tried to move back new op lower bound from 7332606887181926400 to 7332606887059894272. Current Snapshot: MvccSnapshot[applied={T|T < 7332606887181926400}]
I20260923 18:13:48.329192 11394 mvcc.cc:205] Tried to move back new op lower bound from 7332606887181926400 to 7332606887059894272. Current Snapshot: MvccSnapshot[applied={T|T < 7332606887181926400}]
I20260923 18:13:48.336006 10895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:13:48.336208 10895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:13:48.338959 10895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Sending DeleteTablet for 3 replicas of tablet 9a822a90877a49b9ac21395d5cbe2240
I20260923 18:13:48.339804 11023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a822a90877a49b9ac21395d5cbe2240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:48 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:34318
I20260923 18:13:48.339807 11159 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a822a90877a49b9ac21395d5cbe2240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:48 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:44684
I20260923 18:13:48.340376 11402 tablet_replica.cc:330] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de: stopping tablet replica
I20260923 18:13:48.340378 11294 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a822a90877a49b9ac21395d5cbe2240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:48 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:56426
I20260923 18:13:48.340708 11401 tablet_replica.cc:330] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: stopping tablet replica
I20260923 18:13:48.340796 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 10960
I20260923 18:13:48.340889 11403 tablet_replica.cc:330] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: stopping tablet replica
I20260923 18:13:48.341284 11403 raft_consensus.cc:2229] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:13:48.341604 11402 raft_consensus.cc:2229] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:13:48.341751 11403 raft_consensus.cc:2258] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:13:48.342015 11402 raft_consensus.cc:2258] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:13:48.343575 11403 ts_tablet_manager.cc:1905] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:13:48.343580 11402 ts_tablet_manager.cc:1905] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:13:48.345897 11403 ts_tablet_manager.cc:1918] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:13:48.345993 11403 log.cc:1196] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/9a822a90877a49b9ac21395d5cbe2240
I20260923 18:13:48.346283 11403 ts_tablet_manager.cc:1939] T 9a822a90877a49b9ac21395d5cbe2240 P 3b12bf3040754caf8e9269458ea1aa32: Deleting consensus metadata
I20260923 18:13:48.346261 11402 ts_tablet_manager.cc:1918] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:13:48.346345 11402 log.cc:1196] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/9a822a90877a49b9ac21395d5cbe2240
I20260923 18:13:48.346619 11402 ts_tablet_manager.cc:1939] T 9a822a90877a49b9ac21395d5cbe2240 P 6f433495bcd74994bf41c2b7ed29d3de: Deleting consensus metadata
I20260923 18:13:48.347517 10885 catalog_manager.cc:4841] TS 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005): tablet 9a822a90877a49b9ac21395d5cbe2240 (table test-table [id=67660519136747d799215b85215e3b01]) successfully deleted
I20260923 18:13:48.347724 10883 catalog_manager.cc:4841] TS 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813): tablet 9a822a90877a49b9ac21395d5cbe2240 (table test-table [id=67660519136747d799215b85215e3b01]) successfully deleted
W20260923 18:13:48.348285 10884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:33403 (error 108)
I20260923 18:13:48.348536 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 11095
I20260923 18:13:48.349320 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:13:48.349036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:33403 (local address 127.0.0.1:34338)
0923 18:13:48.349170 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 18:13:48.349237 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:33403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 18:13:48.349485 10884 catalog_manager.cc:4584] TS c163e090e5ac4b45a98da3fc9f72baec (127.27.75.193:33403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a822a90877a49b9ac21395d5cbe2240: Network error: Client connection negotiation failed: client connection to 127.27.75.193:33403: connect: Connection refused (error 111)
I20260923 18:13:48.354653 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 11231
I20260923 18:13:48.359673 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 10866
I20260923 18:14:08.365628 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42163
--webserver_interface=127.27.75.254
--webserver_port=46177
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:42163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:08.477417 11405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:08.477708 11405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:08.477751 11405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:08.481914 11405 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 18:14:08.481971 11405 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 18:14:08.481986 11405 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 18:14:08.481998 11405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:14:08.482016 11405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:08.482033 11405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:08.482052 11405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:14:08.482080 11405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:14:08.487540 11405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:42163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=46177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:08.488674 11405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:08.489758 11405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:08.502544 11411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:08.502554 11412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:08.502607 11405 server_base.cc:1034] running on GCE node
W20260923 18:14:08.502542 11414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:08.503355 11405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:08.504588 11405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:08.505780 11405 hybrid_clock.cc:648] HybridClock initialized: now 1790187248505748 us; error 45 us; skew 500 ppm
Sep 23 18:14:08 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187248, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:08.508939 11405 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:14:08.510006 11405 webserver.cc:460] Webserver started at http://127.27.75.254:46177/ using document root <none> and password file <none>
I20260923 18:14:08.510627 11405 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:08.510707 11405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:08.514948 11405 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260923 18:14:08.517242 11421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:08.518674 11405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:14:08.518877 11405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "6ed3020abbec41848b2ce1096718043a"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "d9206c22aa4a65b8d459d479483ffbbc"
server_key_iv: "16ef05c4e1b6ffeee2701b3f4ab4d56a"
server_key_version: "encryptionkey@0"
I20260923 18:14:08.519517 11405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:08.538702 11405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:08.541860 11405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:08.542054 11405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:08.549096 11405 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:42163
I20260923 18:14:08.549427 11474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:42163 every 8 connection(s)
I20260923 18:14:08.550251 11405 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:14:08.552211 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 11405
I20260923 18:14:08.552693 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:33403
--local_ip_for_outbound_sockets=127.27.75.193
--tserver_master_addrs=127.27.75.254:42163
--webserver_port=35747
--webserver_interface=127.27.75.193
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:14:08.555514 11475 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:14:08.557376 11475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Bootstrap starting.
I20260923 18:14:08.574903 11475 log.cc:824] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:08.584237 11475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:14:08.584613 11475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Bootstrap complete.
I20260923 18:14:08.588640 11475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:14:08.589083 11475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed3020abbec41848b2ce1096718043a, State: Initialized, Role: FOLLOWER
I20260923 18:14:08.589646 11475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:14:08.589772 11475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:14:08.589848 11475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:14:08.589952 11475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 1 FOLLOWER]: Advancing to term 2
I20260923 18:14:08.592111 11475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:14:08.592476 11475 leader_election.cc:302] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ed3020abbec41848b2ce1096718043a; no voters:
I20260923 18:14:08.592782 11475 leader_election.cc:288] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 18:14:08.592900 11479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 2 FOLLOWER]: Leader election won for term 2
I20260923 18:14:08.593103 11479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 2 LEADER]: Becoming Leader. State: Replica: 6ed3020abbec41848b2ce1096718043a, State: Running, Role: LEADER
I20260923 18:14:08.593451 11479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:14:08.594151 11475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:14:08.594483 11479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ed3020abbec41848b2ce1096718043a. Latest consensus state: current_term: 2 leader_uuid: "6ed3020abbec41848b2ce1096718043a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } } }
I20260923 18:14:08.594600 11479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:08.594964 11480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6ed3020abbec41848b2ce1096718043a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } } }
I20260923 18:14:08.595140 11480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:08.595029 11483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:14:08.601516 11483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:14:08.602306 11483 catalog_manager.cc:1259] Loaded cluster ID: a5ca4600304649c3ad3a084a8ce7361b
I20260923 18:14:08.602370 11483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:14:08.604197 11483 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:14:08.604918 11483 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Loaded TSK: 0
I20260923 18:14:08.617323 11483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Generated new TSK 1
I20260923 18:14:08.617642 11483 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:14:08.701318 11477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:08.701705 11477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:08.701774 11477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:08.707471 11477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:08.707566 11477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:08.707633 11477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:14:08.713567 11477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:33403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=35747
--tserver_master_addrs=127.27.75.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:08.714682 11477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:08.715710 11477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:08.725721 11503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:08.725729 11505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:08.726038 11477 server_base.cc:1034] running on GCE node
W20260923 18:14:08.726064 11502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:08.726585 11477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:08.727402 11477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:08.728601 11477 hybrid_clock.cc:648] HybridClock initialized: now 1790187248728573 us; error 39 us; skew 500 ppm
Sep 23 18:14:08 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187248, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:08.732172 11477 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:14:08.733395 11477 webserver.cc:460] Webserver started at http://127.27.75.193:35747/ using document root <none> and password file <none>
I20260923 18:14:08.734112 11477 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:08.734221 11477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:08.739593 11477 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 18:14:08.742447 11512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:08.744184 11477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:08.744338 11477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "c163e090e5ac4b45a98da3fc9f72baec"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "934b34b6ce2fd570119870b2ede8584f"
server_key_iv: "de7f0c28f0108a696fefe61e97795b61"
server_key_version: "encryptionkey@0"
I20260923 18:14:08.745110 11477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:08.790159 11477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:08.793903 11477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:08.794075 11477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:08.794723 11477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:08.795847 11519 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 18:14:08.799997 11477 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 18:14:08.800078 11477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260923 18:14:08.800151 11477 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 18:14:08.801985 11477 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 18:14:08.802037 11477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260923 18:14:08.802373 11519 tablet_bootstrap.cc:492] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Bootstrap starting.
I20260923 18:14:08.811915 11477 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:33403
I20260923 18:14:08.812012 11627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:33403 every 8 connection(s)
I20260923 18:14:08.812964 11477 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:14:08.820815 11519 log.cc:824] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:08.821048 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 11477
I20260923 18:14:08.821539 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:41813
--local_ip_for_outbound_sockets=127.27.75.194
--tserver_master_addrs=127.27.75.254:42163
--webserver_port=35449
--webserver_interface=127.27.75.194
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:14:08 dist-test-slave-wsx5 krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187248, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:08.829164 11519 tablet_bootstrap.cc:492] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:14:08.829584 11519 tablet_bootstrap.cc:492] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Bootstrap complete.
I20260923 18:14:08.830754 11519 ts_tablet_manager.cc:1397] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.012s
I20260923 18:14:08.832645 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:08.815464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:60799)
0923 18:14:08.816137 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:08.816197 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:08.817337 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:08.818713 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:08.818723 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:08.819262 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:08.819927 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:08.819949 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:08.821578 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:08.821582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:08.822968 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:08.822983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:08.823821 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:08.823825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:08.824021 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:08.824617 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:08.824636 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:08.826453 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:08.829013 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:08.829028 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:08.829040 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:08.829408 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:08.829838 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:08.829841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:08.829844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:08.830001 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:08.830764 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:08.830771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:08.831124 (+ 353us) client_negotiation.cc:769] Sending connection context
0923 18:14:08.831311 (+ 187us) client_negotiation.cc:240] Negotiation successful
0923 18:14:08.831645 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":140,"threads_started":1}
I20260923 18:14:08.832851 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:08.815475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:60799 (local address 127.27.75.254:42163)
0923 18:14:08.815881 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:08.815903 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:08.816328 (+ 425us) server_negotiation.cc:407] Connection header received
0923 18:14:08.817826 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:08.817838 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:08.818234 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:08.818546 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:08.820156 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:08.821381 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:08.823142 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:08.823666 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:08.823833 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:08.826705 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:08.826712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:08.826732 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:08.826770 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:08.828863 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:08.829554 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:08.829560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:08.829566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:08.829733 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:08.830127 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:08.830131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:08.830133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:08.830580 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:08.830709 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:08.831501 (+ 792us) server_negotiation.cc:299] Negotiation successful
0923 18:14:08.831829 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":132,"threads_started":1}
I20260923 18:14:08.833722 11519 raft_consensus.cc:348] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } }
I20260923 18:14:08.834260 11519 raft_consensus.cc:729] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c163e090e5ac4b45a98da3fc9f72baec, State: Initialized, Role: FOLLOWER
I20260923 18:14:08.834748 11519 consensus_queue.cc:260] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } }
I20260923 18:14:08.835753 11628 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42163
I20260923 18:14:08.835783 11519 ts_tablet_manager.cc:1428] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 18:14:08.836004 11628 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:08.836563 11628 heartbeater.cc:511] Master 127.27.75.254:42163 requested a full tablet report, sending...
I20260923 18:14:08.839380 11436 ts_manager.cc:194] Registered new tserver with Master: c163e090e5ac4b45a98da3fc9f72baec (127.27.75.193:33403)
I20260923 18:14:08.842967 11436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:60799
I20260923 18:14:08.845830 11628 heartbeater.cc:503] Master 127.27.75.254:42163 was elected leader, sending a full tablet report...
Sep 23 18:14:08 dist-test-slave-wsx5 krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187248, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.193@KRBTEST.COM
I20260923 18:14:08.853943 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:08.842479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.27.75.193:33403)
0923 18:14:08.842900 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:08.842916 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:08.843329 (+ 413us) server_negotiation.cc:407] Connection header received
0923 18:14:08.843531 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:08.843536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:08.843756 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:08.843900 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:08.844649 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:08.845748 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:08.847032 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:08.847503 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:08.847587 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:08.850134 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:08.850143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:08.850154 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:08.850178 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:08.851783 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:08.852359 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:08.852364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:08.852365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:08.852450 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:08.852768 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:08.852771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:08.852772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:08.853160 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:08.853280 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:08.853647 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 18:14:08.853798 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":108,"threads_started":1}
I20260923 18:14:08.853976 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:08.842371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:33403 (local address 127.0.0.1:40234)
0923 18:14:08.843170 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:08.843219 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:08.843371 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:08.843928 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:08.843934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:08.844375 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:08.844510 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:08.844518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:08.845870 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:08.845874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:08.846883 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:08.846903 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:08.847610 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:08.847613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:08.847720 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:08.848318 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:08.848333 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:08.849981 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:08.851914 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:08.851925 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:08.851931 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:08.852242 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:08.852536 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:08.852539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:08.852541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:08.852646 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:08.853285 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:08.853291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:08.853430 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 18:14:08.853574 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 18:14:08.853788 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":126,"threads_started":1}
I20260923 18:14:08.855022 11561 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a822a90877a49b9ac21395d5cbe2240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:13:48 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:40234
I20260923 18:14:08.855911 11637 tablet_replica.cc:330] stopping tablet replica
I20260923 18:14:08.856238 11637 raft_consensus.cc:2229] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:08.856495 11637 raft_consensus.cc:2258] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:08.857586 11637 ts_tablet_manager.cc:1905] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:08.862517 11637 ts_tablet_manager.cc:1918] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:08.862653 11637 log.cc:1196] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal/wals/9a822a90877a49b9ac21395d5cbe2240
I20260923 18:14:08.862920 11637 ts_tablet_manager.cc:1939] T 9a822a90877a49b9ac21395d5cbe2240 P c163e090e5ac4b45a98da3fc9f72baec: Deleting consensus metadata
I20260923 18:14:08.865342 11424 catalog_manager.cc:4841] TS c163e090e5ac4b45a98da3fc9f72baec (127.27.75.193:33403): tablet 9a822a90877a49b9ac21395d5cbe2240 (table test-table [id=67660519136747d799215b85215e3b01]) successfully deleted
W20260923 18:14:08.947583 11633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:08.947870 11633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:08.947916 11633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:08.952142 11633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:08.952217 11633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:08.952279 11633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:14:08.958132 11633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:41813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=35449
--tserver_master_addrs=127.27.75.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:08.959575 11633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:08.960655 11633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:14:08.970666 11633 server_base.cc:1034] running on GCE node
W20260923 18:14:08.970631 11645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:08.970644 11642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:08.970642 11643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:08.971429 11633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:08.972078 11633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:08.973227 11633 hybrid_clock.cc:648] HybridClock initialized: now 1790187248973205 us; error 35 us; skew 500 ppm
Sep 23 18:14:08 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187248, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:08.976012 11633 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:14:08.976924 11633 webserver.cc:460] Webserver started at http://127.27.75.194:35449/ using document root <none> and password file <none>
I20260923 18:14:08.977516 11633 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:08.977597 11633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:08.981936 11633 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:14:08.984244 11652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:08.985776 11633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:08.985899 11633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "6f433495bcd74994bf41c2b7ed29d3de"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "8b019351162935cbae572ce7541ffb11"
server_key_iv: "551427aa5ba6c6fff6fee04341f8f382"
server_key_version: "encryptionkey@0"
I20260923 18:14:08.986488 11633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:09.003007 11633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:09.006263 11633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:09.006450 11633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:09.007112 11633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:09.007953 11633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:09.008000 11633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:09.008045 11633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:09.008073 11633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:09.016580 11633 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:41813
I20260923 18:14:09.016613 11765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:41813 every 8 connection(s)
I20260923 18:14:09.017668 11633 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:14:09.019461 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 11633
I20260923 18:14:09.019898 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:41005
--local_ip_for_outbound_sockets=127.27.75.195
--tserver_master_addrs=127.27.75.254:42163
--webserver_port=34047
--webserver_interface=127.27.75.195
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 18:14:09 dist-test-slave-wsx5 krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187248, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:09.033459 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.019643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:39683 (local address 127.27.75.254:42163)
0923 18:14:09.019799 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:09.019804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:09.020830 (+ 1026us) server_negotiation.cc:407] Connection header received
0923 18:14:09.022266 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:09.022269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:09.022322 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:09.022408 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:09.024087 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.024669 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.026555 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.026797 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.026886 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.029974 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:09.029977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:09.029980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:09.030008 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:09.031349 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:09.031867 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:09.031870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:09.031871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:09.031959 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:09.032286 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:09.032288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:09.032289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:09.032482 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:09.032569 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:09.033103 (+ 534us) server_negotiation.cc:299] Negotiation successful
0923 18:14:09.033284 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:14:09.034348 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.020085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:39683)
0923 18:14:09.020642 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:09.020703 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:09.022046 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:09.022642 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:09.022652 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:09.023209 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:09.023909 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.023932 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.024829 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.024834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.026411 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.026425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.026975 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.026980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.027195 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.027854 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:09.027877 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:09.029718 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:09.031471 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:09.031477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:09.031496 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:09.031759 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:09.032042 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:09.032045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:09.032047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:09.032190 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:09.032580 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:09.032586 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:09.032856 (+ 270us) client_negotiation.cc:769] Sending connection context
0923 18:14:09.033027 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 18:14:09.033429 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":125,"threads_started":1}
I20260923 18:14:09.035655 11766 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42163
I20260923 18:14:09.035959 11766 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:09.036471 11766 heartbeater.cc:511] Master 127.27.75.254:42163 requested a full tablet report, sending...
I20260923 18:14:09.037618 11436 ts_manager.cc:194] Registered new tserver with Master: 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813)
I20260923 18:14:09.038415 11436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:39683
W20260923 18:14:09.136359 11769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:09.136835 11769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:09.136924 11769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:09.141292 11769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:09.141361 11769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:09.141438 11769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:14:09.147008 11769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:41005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=34047
--tserver_master_addrs=127.27.75.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:09.148065 11769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:09.149042 11769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:09.158892 11774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:09.158964 11777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:09.158896 11775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:09.159116 11769 server_base.cc:1034] running on GCE node
I20260923 18:14:09.159708 11769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:09.160449 11769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:09.161607 11769 hybrid_clock.cc:648] HybridClock initialized: now 1790187249161588 us; error 36 us; skew 500 ppm
Sep 23 18:14:09 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187249, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:09.164434 11769 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:14:09.165412 11769 webserver.cc:460] Webserver started at http://127.27.75.195:34047/ using document root <none> and password file <none>
I20260923 18:14:09.165962 11769 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:09.166042 11769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:09.170063 11769 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:14:09.172077 11784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:09.173472 11769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:14:09.173588 11769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "3b12bf3040754caf8e9269458ea1aa32"
format_stamp: "Formatted at 2026-09-23 18:13:48 on dist-test-slave-wsx5"
server_key: "71841c4e3aa6477ee55a2ce255216467"
server_key_iv: "ba3991cf09c342686774c63c2c0293ff"
server_key_version: "encryptionkey@0"
I20260923 18:14:09.174139 11769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:09.200942 11769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:09.204108 11769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:09.204295 11769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:09.204926 11769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:09.205785 11769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:09.205843 11769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:09.205899 11769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:09.205927 11769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:09.214588 11769 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:41005
I20260923 18:14:09.214613 11898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:41005 every 8 connection(s)
I20260923 18:14:09.215756 11769 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:14:09.217270 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 11769
Sep 23 18:14:09 dist-test-slave-wsx5 krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187249, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:09.226745 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.217916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.27.75.254:42163)
0923 18:14:09.218122 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:09.218126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:09.218300 (+ 174us) server_negotiation.cc:407] Connection header received
0923 18:14:09.218435 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:09.218438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:09.218498 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:09.218599 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:09.219504 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.220073 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.221245 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.221525 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.221621 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.223249 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:09.223253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:09.223256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:09.223288 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:09.224993 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:09.225410 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:09.225413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:09.225415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:09.225498 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:09.225741 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:09.225744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:09.225746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:09.225943 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:09.226157 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:09.226321 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 18:14:09.226441 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260923 18:14:09.230335 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.217735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:58245 (local address 127.27.75.254:42163)
0923 18:14:09.217902 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:09.217907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:09.218984 (+ 1077us) server_negotiation.cc:407] Connection header received
0923 18:14:09.220405 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:09.220408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:09.220458 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:09.220503 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:09.222003 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.222617 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.223951 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.224264 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.224374 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.227090 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:09.227093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:09.227096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:09.227120 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:09.228358 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:09.228851 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:09.228854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:09.228856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:09.228933 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:09.229300 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:09.229303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:09.229305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:09.229474 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:09.229567 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:09.230062 (+ 495us) server_negotiation.cc:299] Negotiation successful
0923 18:14:09.230182 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 18:14:09.231171 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.218267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:58245)
0923 18:14:09.218822 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:09.218880 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:09.220193 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:09.220721 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:09.220731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:09.221253 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:09.221848 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.221869 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.222755 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.222759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.223834 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.223844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.224436 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.224441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.224676 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.225310 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:09.225326 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:09.226947 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:09.228476 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:09.228482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:09.228500 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:09.228759 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:09.229024 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:09.229027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:09.229030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:09.229215 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:09.229567 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:09.229574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:09.229828 (+ 254us) client_negotiation.cc:769] Sending connection context
0923 18:14:09.229989 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 18:14:09.230329 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":151,"threads_started":1}
I20260923 18:14:09.232353 11899 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42163
I20260923 18:14:09.232604 11899 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:09.233155 11899 heartbeater.cc:511] Master 127.27.75.254:42163 requested a full tablet report, sending...
I20260923 18:14:09.234390 11436 ts_manager.cc:194] Registered new tserver with Master: 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005)
I20260923 18:14:09.234992 11436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:58245
I20260923 18:14:09.240573 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:14:09.244707 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.241272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.27.75.254:42163)
0923 18:14:09.241386 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:09.241390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:09.241590 (+ 200us) server_negotiation.cc:407] Connection header received
0923 18:14:09.241784 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:09.241786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:09.241892 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:09.241989 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:14:09.242223 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.242726 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.243602 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.243819 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.243900 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.244221 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:14:09.244366 (+ 145us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 18:14:09.244591 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55228: token expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260923 18:14:09.244796 11630 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55228: token expired
W20260923 18:14:09.245359 11366 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.75.254:42163)
I20260923 18:14:09.252610 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.245723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.27.75.254:42163)
0923 18:14:09.245850 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:09.245853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:09.245866 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:14:09.245957 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:09.245960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:09.246002 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:09.246066 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:09.246671 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.247163 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.247941 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.248160 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.248245 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.249557 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:09.249561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:09.249564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:09.249591 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:09.251076 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:09.251484 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:09.251487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:09.251488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:09.251572 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:09.251791 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:09.251793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:09.251794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:09.252011 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:09.252124 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:09.252331 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 18:14:09.252442 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:14:09.255259 11436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:14:09.258581 11436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:14:09.266335 11561 tablet_service.cc:1467] Processing CreateTablet for tablet e86bbe5c4b71422da5b40012b7c574a1 (DEFAULT_TABLE table=test-table [id=e1f356a6cfd44163bdad629b2d897a51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:09.267066 11561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86bbe5c4b71422da5b40012b7c574a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:09.270063 11519 tablet_bootstrap.cc:492] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: Bootstrap starting.
I20260923 18:14:09.270644 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.265191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:41005 (local address 127.0.0.1:38482)
0923 18:14:09.265311 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:09.265327 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:09.265414 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:09.266271 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:09.266274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:09.266292 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:09.266407 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.266414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.267721 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.267724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.268960 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.268972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.270180 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.270184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.270308 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.270356 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 18:14:09.270412 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 18:14:09.270511 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 18:14:09.271024 11519 tablet_bootstrap.cc:654] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:09.271080 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.265669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:41813 (local address 127.0.0.1:54284)
0923 18:14:09.265927 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:09.265940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:09.266038 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:09.266986 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:09.266989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:09.267009 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:09.267098 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.267103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.268527 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.268531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.269564 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.269574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.270690 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.270694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.270780 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.270802 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:14:09.270881 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 18:14:09.270955 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":101,"threads_started":1}
I20260923 18:14:09.271255 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.265390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.27.75.195:41005)
0923 18:14:09.265767 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:09.265778 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:09.265803 (+ 25us) server_negotiation.cc:407] Connection header received
0923 18:14:09.265888 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:09.265894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:09.266137 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:09.266282 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:09.266534 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.267593 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.269099 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.270038 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.270800 (+ 762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.270954 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:09.271010 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 18:14:09.271115 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":178,"threads_started":1}
I20260923 18:14:09.271564 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.266090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.27.75.194:41813)
0923 18:14:09.266508 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:09.266520 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:09.266545 (+ 25us) server_negotiation.cc:407] Connection header received
0923 18:14:09.266622 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:09.266627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:09.266819 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:09.266990 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:09.267280 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.268399 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.269683 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.270365 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.270479 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.270627 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:09.271299 (+ 672us) server_negotiation.cc:299] Negotiation successful
0923 18:14:09.271412 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":112,"threads_started":1}
I20260923 18:14:09.272774 11519 tablet_bootstrap.cc:492] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: No bootstrap required, opened a new log
I20260923 18:14:09.272890 11519 ts_tablet_manager.cc:1397] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260923 18:14:09.273379 11519 raft_consensus.cc:348] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:09.273504 11519 raft_consensus.cc:374] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:09.273542 11519 raft_consensus.cc:729] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c163e090e5ac4b45a98da3fc9f72baec, State: Initialized, Role: FOLLOWER
I20260923 18:14:09.273701 11519 consensus_queue.cc:260] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:09.274564 11519 ts_tablet_manager.cc:1428] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 18:14:09.274210 11833 tablet_service.cc:1467] Processing CreateTablet for tablet e86bbe5c4b71422da5b40012b7c574a1 (DEFAULT_TABLE table=test-table [id=e1f356a6cfd44163bdad629b2d897a51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:09.274433 11700 tablet_service.cc:1467] Processing CreateTablet for tablet e86bbe5c4b71422da5b40012b7c574a1 (DEFAULT_TABLE table=test-table [id=e1f356a6cfd44163bdad629b2d897a51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:09.275336 11700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86bbe5c4b71422da5b40012b7c574a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:09.275377 11833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86bbe5c4b71422da5b40012b7c574a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:09.282711 11913 tablet_bootstrap.cc:492] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de: Bootstrap starting.
I20260923 18:14:09.282711 11912 tablet_bootstrap.cc:492] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32: Bootstrap starting.
I20260923 18:14:09.285219 11912 tablet_bootstrap.cc:654] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:09.285260 11913 tablet_bootstrap.cc:654] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:09.286002 11913 log.cc:824] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:09.286002 11912 log.cc:824] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:09.288048 11913 tablet_bootstrap.cc:492] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de: No bootstrap required, opened a new log
I20260923 18:14:09.288245 11913 ts_tablet_manager.cc:1397] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:14:09.288430 11912 tablet_bootstrap.cc:492] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32: No bootstrap required, opened a new log
I20260923 18:14:09.288617 11912 ts_tablet_manager.cc:1397] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:14:09.291743 11913 raft_consensus.cc:348] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:09.292057 11913 raft_consensus.cc:374] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:09.291867 11912 raft_consensus.cc:348] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:09.292124 11913 raft_consensus.cc:729] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f433495bcd74994bf41c2b7ed29d3de, State: Initialized, Role: FOLLOWER
I20260923 18:14:09.292143 11912 raft_consensus.cc:374] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:09.292200 11912 raft_consensus.cc:729] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b12bf3040754caf8e9269458ea1aa32, State: Initialized, Role: FOLLOWER
I20260923 18:14:09.292645 11912 consensus_queue.cc:260] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:09.292649 11913 consensus_queue.cc:260] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:09.293596 11766 heartbeater.cc:503] Master 127.27.75.254:42163 was elected leader, sending a full tablet report...
I20260923 18:14:09.293838 11913 ts_tablet_manager.cc:1428] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 18:14:09.293844 11912 ts_tablet_manager.cc:1428] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260923 18:14:09.294188 11899 heartbeater.cc:503] Master 127.27.75.254:42163 was elected leader, sending a full tablet report...
W20260923 18:14:09.314210 11629 tablet.cc:2369] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:14:09.382848 11634 raft_consensus.cc:482] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:09.383147 11634 raft_consensus.cc:504] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:09.384248 11634 leader_election.cc:288] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005), 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813)
I20260923 18:14:09.388036 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.384631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:38737 (local address 127.27.75.194:41813)
0923 18:14:09.384759 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:09.384763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:09.385111 (+ 348us) server_negotiation.cc:407] Connection header received
0923 18:14:09.385258 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:09.385261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:09.385307 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:09.385369 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:09.385584 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.386060 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.387119 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.387500 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.387610 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.387662 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:09.387832 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 18:14:09.387905 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:14:09.388046 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.384632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:45267 (local address 127.27.75.195:41005)
0923 18:14:09.384759 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:09.384764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:09.384961 (+ 197us) server_negotiation.cc:407] Connection header received
0923 18:14:09.385156 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:09.385159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:09.385203 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:09.385265 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:09.385503 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.385991 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.387116 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.387502 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.387607 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.387662 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:09.387840 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 18:14:09.387909 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:14:09.388088 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.384535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:41813 (local address 127.27.75.193:38737)
0923 18:14:09.384986 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:09.384997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:09.385157 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:09.385393 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:09.385395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:09.385411 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:09.385493 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.385497 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.386159 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.386162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.387003 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.387019 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.387611 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.387614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.387706 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.387742 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 18:14:09.387808 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 18:14:09.387886 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":112,"thread_start_us":75,"threads_started":1}
I20260923 18:14:09.388098 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.384499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:41005 (local address 127.27.75.193:45267)
0923 18:14:09.384868 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:09.384880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:09.385001 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:09.385286 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:09.385289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:09.385306 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:09.385407 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.385412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.386128 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.386131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.387000 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:09.387008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.387607 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.387610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:09.387709 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.387751 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 18:14:09.387813 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 18:14:09.387890 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":13,"thread_start_us":86,"threads_started":1}
I20260923 18:14:09.388803 11720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86bbe5c4b71422da5b40012b7c574a1" candidate_uuid: "c163e090e5ac4b45a98da3fc9f72baec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f433495bcd74994bf41c2b7ed29d3de" is_pre_election: true
I20260923 18:14:09.389112 11720 raft_consensus.cc:2454] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c163e090e5ac4b45a98da3fc9f72baec in term 0.
I20260923 18:14:09.389509 11853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86bbe5c4b71422da5b40012b7c574a1" candidate_uuid: "c163e090e5ac4b45a98da3fc9f72baec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b12bf3040754caf8e9269458ea1aa32" is_pre_election: true
I20260923 18:14:09.389665 11514 leader_election.cc:302] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f433495bcd74994bf41c2b7ed29d3de, c163e090e5ac4b45a98da3fc9f72baec; no voters:
I20260923 18:14:09.389750 11853 raft_consensus.cc:2454] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c163e090e5ac4b45a98da3fc9f72baec in term 0.
I20260923 18:14:09.389936 11634 raft_consensus.cc:2781] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:14:09.390023 11634 raft_consensus.cc:482] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:14:09.390076 11634 raft_consensus.cc:3037] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:09.391165 11634 raft_consensus.cc:504] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:09.391516 11634 leader_election.cc:288] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 1 election: Requested vote from peers 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005), 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813)
I20260923 18:14:09.391865 11853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86bbe5c4b71422da5b40012b7c574a1" candidate_uuid: "c163e090e5ac4b45a98da3fc9f72baec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b12bf3040754caf8e9269458ea1aa32"
I20260923 18:14:09.391939 11720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86bbe5c4b71422da5b40012b7c574a1" candidate_uuid: "c163e090e5ac4b45a98da3fc9f72baec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f433495bcd74994bf41c2b7ed29d3de"
I20260923 18:14:09.391990 11853 raft_consensus.cc:3037] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:09.392088 11720 raft_consensus.cc:3037] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:09.393149 11853 raft_consensus.cc:2454] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c163e090e5ac4b45a98da3fc9f72baec in term 1.
I20260923 18:14:09.393291 11720 raft_consensus.cc:2454] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c163e090e5ac4b45a98da3fc9f72baec in term 1.
I20260923 18:14:09.393465 11516 leader_election.cc:302] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b12bf3040754caf8e9269458ea1aa32, c163e090e5ac4b45a98da3fc9f72baec; no voters:
I20260923 18:14:09.393635 11634 raft_consensus.cc:2781] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:09.393812 11634 raft_consensus.cc:686] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 LEADER]: Becoming Leader. State: Replica: c163e090e5ac4b45a98da3fc9f72baec, State: Running, Role: LEADER
I20260923 18:14:09.394104 11634 consensus_queue.cc:237] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:09.397110 11438 catalog_manager.cc:5495] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec reported cstate change: term changed from 0 to 1, leader changed from <none> to c163e090e5ac4b45a98da3fc9f72baec (127.27.75.193). New cstate: current_term: 1 leader_uuid: "c163e090e5ac4b45a98da3fc9f72baec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } health_report { overall_health: HEALTHY } } }
I20260923 18:14:09.430402 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.426888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.27.75.193:33403)
0923 18:14:09.427169 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:09.427176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:09.427188 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:14:09.427264 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:09.427266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:09.427319 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:09.427419 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:14:09.427633 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.428176 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.428996 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:09.429241 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:09.429321 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:09.429567 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:14:09.430056 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:14:09.430153 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:09.430205 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 18:14:09.430260 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":114,"threads_started":1}
I20260923 18:14:09.434509 11720 raft_consensus.cc:1261] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de [term 1 FOLLOWER]: Refusing update from remote peer c163e090e5ac4b45a98da3fc9f72baec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:09.434508 11853 raft_consensus.cc:1261] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32 [term 1 FOLLOWER]: Refusing update from remote peer c163e090e5ac4b45a98da3fc9f72baec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:09.435231 11921 consensus_queue.cc:1035] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:09.435410 11634 consensus_queue.cc:1035] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:09.443507 11925 mvcc.cc:205] Tried to move back new op lower bound from 7332606973677068288 to 7332606973521027072. Current Snapshot: MvccSnapshot[applied={T|T < 7332606973677068288}]
I20260923 18:14:09.451591 11438 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:09.451821 11438 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:09.454552 11438 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Sending DeleteTablet for 3 replicas of tablet e86bbe5c4b71422da5b40012b7c574a1
I20260923 18:14:09.455291 11561 tablet_service.cc:1514] Processing DeleteTablet for tablet e86bbe5c4b71422da5b40012b7c574a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:09 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:40234
I20260923 18:14:09.455390 11833 tablet_service.cc:1514] Processing DeleteTablet for tablet e86bbe5c4b71422da5b40012b7c574a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:09 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:38482
I20260923 18:14:09.455437 11700 tablet_service.cc:1514] Processing DeleteTablet for tablet e86bbe5c4b71422da5b40012b7c574a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:09 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:54284
I20260923 18:14:09.455926 11934 tablet_replica.cc:330] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32: stopping tablet replica
I20260923 18:14:09.456254 11935 tablet_replica.cc:330] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de: stopping tablet replica
I20260923 18:14:09.456284 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 11477
I20260923 18:14:09.456272 11934 raft_consensus.cc:2229] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:09.456626 11934 raft_consensus.cc:2258] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:09.457178 11935 raft_consensus.cc:2229] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:09.457651 11935 raft_consensus.cc:2258] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:09.458715 11934 ts_tablet_manager.cc:1905] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:09.458982 11935 ts_tablet_manager.cc:1905] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:09.461198 11934 ts_tablet_manager.cc:1918] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:09.461297 11934 log.cc:1196] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/e86bbe5c4b71422da5b40012b7c574a1
I20260923 18:14:09.461597 11934 ts_tablet_manager.cc:1939] T e86bbe5c4b71422da5b40012b7c574a1 P 3b12bf3040754caf8e9269458ea1aa32: Deleting consensus metadata
I20260923 18:14:09.461676 11935 ts_tablet_manager.cc:1918] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:09.461772 11935 log.cc:1196] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/e86bbe5c4b71422da5b40012b7c574a1
I20260923 18:14:09.462143 11935 ts_tablet_manager.cc:1939] T e86bbe5c4b71422da5b40012b7c574a1 P 6f433495bcd74994bf41c2b7ed29d3de: Deleting consensus metadata
I20260923 18:14:09.462913 11425 catalog_manager.cc:4841] TS 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005): tablet e86bbe5c4b71422da5b40012b7c574a1 (table test-table [id=e1f356a6cfd44163bdad629b2d897a51]) successfully deleted
I20260923 18:14:09.463456 11423 catalog_manager.cc:4841] TS 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813): tablet e86bbe5c4b71422da5b40012b7c574a1 (table test-table [id=e1f356a6cfd44163bdad629b2d897a51]) successfully deleted
W20260923 18:14:09.463913 11424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:33403 (error 108)
I20260923 18:14:09.464740 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 11633
I20260923 18:14:09.465013 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:09.464636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:33403 (local address 127.0.0.1:40256)
0923 18:14:09.464798 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:09.464878 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:33403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260923 18:14:09.466506 11424 catalog_manager.cc:4584] TS c163e090e5ac4b45a98da3fc9f72baec (127.27.75.193:33403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e86bbe5c4b71422da5b40012b7c574a1: Network error: Client connection negotiation failed: client connection to 127.27.75.193:33403: connect: Connection refused (error 111)
I20260923 18:14:09.470680 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 11769
I20260923 18:14:09.476249 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 11405
I20260923 18:14:29.482568 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42163
--webserver_interface=127.27.75.254
--webserver_port=46177
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:42163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:29.603072 11937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:29.603368 11937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:29.603410 11937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:29.607512 11937 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 18:14:29.607583 11937 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 18:14:29.607599 11937 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 18:14:29.607611 11937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:14:29.607630 11937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:29.607646 11937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:29.607661 11937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:14:29.607676 11937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:14:29.613483 11937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:42163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=46177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:29.614603 11937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:29.615641 11937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:29.624927 11945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:29.625000 11937 server_base.cc:1034] running on GCE node
W20260923 18:14:29.624939 11943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:29.624954 11942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:29.625826 11937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:29.627004 11937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:29.628187 11937 hybrid_clock.cc:648] HybridClock initialized: now 1790187269628167 us; error 34 us; skew 500 ppm
Sep 23 18:14:29 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187269, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:29.638602 11937 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:14:29.639668 11937 webserver.cc:460] Webserver started at http://127.27.75.254:46177/ using document root <none> and password file <none>
I20260923 18:14:29.640288 11937 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:29.640362 11937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:29.644361 11937 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:14:29.646715 11952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:29.648046 11937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:14:29.648183 11937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "6ed3020abbec41848b2ce1096718043a"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "d9206c22aa4a65b8d459d479483ffbbc"
server_key_iv: "16ef05c4e1b6ffeee2701b3f4ab4d56a"
server_key_version: "encryptionkey@0"
I20260923 18:14:29.648739 11937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:29.676656 11937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:29.680092 11937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:29.680294 11937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:29.687460 11937 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:42163
I20260923 18:14:29.687733 12004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:42163 every 8 connection(s)
I20260923 18:14:29.688580 11937 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:14:29.690011 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 11937
I20260923 18:14:29.690443 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:33403
--local_ip_for_outbound_sockets=127.27.75.193
--tserver_master_addrs=127.27.75.254:42163
--webserver_port=35747
--webserver_interface=127.27.75.193
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:14:29.693434 12005 sys_catalog.cc:263] Verifying existing consensus state
I20260923 18:14:29.695237 12005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Bootstrap starting.
I20260923 18:14:29.712878 12005 log.cc:824] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:29.724707 12005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:14:29.725095 12005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: Bootstrap complete.
I20260923 18:14:29.730343 12005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:14:29.730937 12005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed3020abbec41848b2ce1096718043a, State: Initialized, Role: FOLLOWER
I20260923 18:14:29.731537 12005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:14:29.731704 12005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:14:29.731803 12005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:14:29.731907 12005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 2 FOLLOWER]: Advancing to term 3
I20260923 18:14:29.734230 12005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:14:29.734575 12005 leader_election.cc:302] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ed3020abbec41848b2ce1096718043a; no voters:
I20260923 18:14:29.734901 12005 leader_election.cc:288] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 18:14:29.735044 12009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 3 FOLLOWER]: Leader election won for term 3
I20260923 18:14:29.735391 12009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [term 3 LEADER]: Becoming Leader. State: Replica: 6ed3020abbec41848b2ce1096718043a, State: Running, Role: LEADER
I20260923 18:14:29.735775 12009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } }
I20260923 18:14:29.736096 12005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:14:29.737295 12011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ed3020abbec41848b2ce1096718043a. Latest consensus state: current_term: 3 leader_uuid: "6ed3020abbec41848b2ce1096718043a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } } }
I20260923 18:14:29.737454 12011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:29.738447 12010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6ed3020abbec41848b2ce1096718043a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed3020abbec41848b2ce1096718043a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42163 } } }
I20260923 18:14:29.738736 12010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:29.741647 12026 catalog_manager.cc:1259] Loaded cluster ID: a5ca4600304649c3ad3a084a8ce7361b
I20260923 18:14:29.741720 12026 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: loading cluster ID for follower catalog manager: success
I20260923 18:14:29.744333 12026 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ed3020abbec41848b2ce1096718043a: acquiring CA information for follower catalog manager: success
F20260923 18:14:29.745803 12026 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3e2822f9ad google::LogMessage::Fail() at ??:0
@ 0x7f3e2823288a google::LogMessage::SendToLog() at ??:0
@ 0x7f3e2822f4de google::LogMessage::Flush() at ??:0
@ 0x7f3e28230a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3e2aef318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3e32b98989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f3e32b75023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f3e32b7554b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f3e32b6e652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f3e32b6d9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f3e32bb4541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3e2a472512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3e2bab66db start_thread at ??:0
@ 0x7f3e26d2d61f clone at ??:0
W20260923 18:14:29.825709 12007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:29.826030 12007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:29.826095 12007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:29.830235 12007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:29.830305 12007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:29.830395 12007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:14:29.835924 12007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:33403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=35747
--tserver_master_addrs=127.27.75.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:29.837087 12007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:29.838209 12007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:29.848196 12046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:29.848215 12047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:29.848255 12049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:29.848515 12007 server_base.cc:1034] running on GCE node
I20260923 18:14:29.849092 12007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:29.849790 12007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:29.850958 12007 hybrid_clock.cc:648] HybridClock initialized: now 1790187269850930 us; error 48 us; skew 500 ppm
Sep 23 18:14:29 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187269, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:29.853919 12007 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:14:29.855162 12007 webserver.cc:460] Webserver started at http://127.27.75.193:35747/ using document root <none> and password file <none>
I20260923 18:14:29.855966 12007 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:29.856067 12007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:29.861405 12007 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:29.864537 12056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:29.866268 12007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:29.866433 12007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "c163e090e5ac4b45a98da3fc9f72baec"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "934b34b6ce2fd570119870b2ede8584f"
server_key_iv: "de7f0c28f0108a696fefe61e97795b61"
server_key_version: "encryptionkey@0"
I20260923 18:14:29.867272 12007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:29.892030 12007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:29.896283 12007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:29.896517 12007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:29.897264 12007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:29.898710 12064 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 18:14:29.903024 12007 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 18:14:29.903124 12007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260923 18:14:29.903211 12007 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 18:14:29.905758 12007 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 18:14:29.905843 12007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260923 18:14:29.906231 12064 tablet_bootstrap.cc:492] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: Bootstrap starting.
I20260923 18:14:29.916759 12007 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:33403
I20260923 18:14:29.917037 12172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:33403 every 8 connection(s)
I20260923 18:14:29.918107 12007 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:14:29.926507 12064 log.cc:824] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:29.927695 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 12007
I20260923 18:14:29.928161 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:41813
--local_ip_for_outbound_sockets=127.27.75.194
--tserver_master_addrs=127.27.75.254:42163
--webserver_port=35449
--webserver_interface=127.27.75.194
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:14:29.934564 12064 tablet_bootstrap.cc:492] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 18:14:29.935012 12064 tablet_bootstrap.cc:492] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: Bootstrap complete.
I20260923 18:14:29.936307 12064 ts_tablet_manager.cc:1397] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.005s
I20260923 18:14:29.940210 12064 raft_consensus.cc:348] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:29.941020 12064 raft_consensus.cc:729] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c163e090e5ac4b45a98da3fc9f72baec, State: Initialized, Role: FOLLOWER
I20260923 18:14:29.942047 12064 consensus_queue.cc:260] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:29.943398 12064 ts_tablet_manager.cc:1428] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
W20260923 18:14:29.970664 12175 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:14:29.920668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:33795)
0923 18:14:29.921379 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:29.921452 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:29.922595 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:29.969894 (+ 47299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":180,"threads_started":1}
W20260923 18:14:29.971274 12173 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:42163 (0 consecutive failures): Network error: Failed to ping master at 127.27.75.254:42163: Client connection negotiation failed: client connection to 127.27.75.254:42163: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 18:14:29.972080 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:29.971780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:54735)
0923 18:14:29.971935 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:29.972011 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 18:14:29.972898 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:29.972644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:56513)
0923 18:14:29.972764 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:29.972827 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 18:14:29.973095 12173 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 18:14:30.057564 12177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:30.057848 12177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:30.057898 12177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:30.062144 12177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:30.062224 12177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:30.062287 12177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:14:30.068315 12177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:41813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=35449
--tserver_master_addrs=127.27.75.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:30.069509 12177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:30.070539 12177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:30.080577 12184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:30.080577 12183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:30.080646 12186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:30.081053 12177 server_base.cc:1034] running on GCE node
I20260923 18:14:30.081548 12177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:30.082229 12177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:30.083411 12177 hybrid_clock.cc:648] HybridClock initialized: now 1790187270083395 us; error 31 us; skew 500 ppm
Sep 23 18:14:30 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:30.086324 12177 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:14:30.087301 12177 webserver.cc:460] Webserver started at http://127.27.75.194:35449/ using document root <none> and password file <none>
I20260923 18:14:30.087862 12177 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:30.087993 12177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:30.092168 12177 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:14:30.094530 12193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:30.095978 12177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:14:30.096109 12177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "6f433495bcd74994bf41c2b7ed29d3de"
format_stamp: "Formatted at 2026-09-23 18:13:47 on dist-test-slave-wsx5"
server_key: "8b019351162935cbae572ce7541ffb11"
server_key_iv: "551427aa5ba6c6fff6fee04341f8f382"
server_key_version: "encryptionkey@0"
I20260923 18:14:30.096745 12177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:30.124936 12177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:30.128216 12177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:30.128410 12177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:30.129411 12177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:30.130400 12177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:30.130458 12177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:30.130522 12177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:30.130553 12177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:30.140287 12177 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:41813
I20260923 18:14:30.140316 12308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:41813 every 8 connection(s)
I20260923 18:14:30.141501 12177 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:14:30.145766 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:30.143807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:36723)
0923 18:14:30.144391 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:30.144522 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":149,"threads_started":1}
W20260923 18:14:30.146286 12309 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:42163 (0 consecutive failures): Network error: Failed to ping master at 127.27.75.254:42163: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
I20260923 18:14:30.146999 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:30.146744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:40155)
0923 18:14:30.146880 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:30.146937 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 18:14:30.147766 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:30.147543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:42899)
0923 18:14:30.147661 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:30.147710 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 18:14:30.148070 12309 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 18:14:30.148914 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 12177
I20260923 18:14:30.149395 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:41005
--local_ip_for_outbound_sockets=127.27.75.195
--tserver_master_addrs=127.27.75.254:42163
--webserver_port=34047
--webserver_interface=127.27.75.195
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:30.264948 12312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:30.265288 12312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:30.265353 12312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:30.269618 12312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:30.269707 12312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:30.269770 12312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:14:30.275486 12312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:41005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=34047
--tserver_master_addrs=127.27.75.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:30.276595 12312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:30.277675 12312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:30.287739 12318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:30.287720 12317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:30.287837 12312 server_base.cc:1034] running on GCE node
W20260923 18:14:30.287720 12320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:30.288381 12312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:30.289058 12312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:30.290259 12312 hybrid_clock.cc:648] HybridClock initialized: now 1790187270290233 us; error 44 us; skew 500 ppm
Sep 23 18:14:30 dist-test-slave-wsx5 krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187270, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:30.293303 12312 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:14:30.294384 12312 webserver.cc:460] Webserver started at http://127.27.75.195:34047/ using document root <none> and password file <none>
I20260923 18:14:30.294979 12312 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:30.295055 12312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:30.299299 12312 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 18:14:30.301779 12327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:30.303306 12312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:30.303457 12312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "3b12bf3040754caf8e9269458ea1aa32"
format_stamp: "Formatted at 2026-09-23 18:13:48 on dist-test-slave-wsx5"
server_key: "71841c4e3aa6477ee55a2ce255216467"
server_key_iv: "ba3991cf09c342686774c63c2c0293ff"
server_key_version: "encryptionkey@0"
I20260923 18:14:30.304081 12312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:30.316377 12312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:30.319790 12312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:30.319996 12312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:30.320616 12312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:30.321538 12312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:30.321586 12312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:30.321642 12312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:30.321671 12312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:30.330611 12312 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:41005
I20260923 18:14:30.330651 12441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:41005 every 8 connection(s)
I20260923 18:14:30.331681 12312 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:14:30.335799 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:30.334044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:51365)
0923 18:14:30.334539 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:30.334985 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":109,"threads_started":1}
I20260923 18:14:30.336308 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 12312
W20260923 18:14:30.336319 12442 heartbeater.cc:644] Failed to heartbeat to 127.27.75.254:42163 (0 consecutive failures): Network error: Failed to ping master at 127.27.75.254:42163: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
I20260923 18:14:30.337073 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:30.336735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:45441)
0923 18:14:30.336933 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:30.336992 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 18:14:30.337755 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:30.337568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:45897)
0923 18:14:30.337661 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:30.337703 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 18:14:30.337981 12442 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 18:14:30.974243 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:30.973761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:54395)
0923 18:14:30.974050 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:30.974117 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
I20260923 18:14:31.149281 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:31.148696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:38647)
0923 18:14:31.149043 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:31.149109 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":77,"threads_started":1}
I20260923 18:14:31.190222 12448 raft_consensus.cc:482] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:31.190534 12448 raft_consensus.cc:504] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:31.191623 12448 leader_election.cc:288] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005), 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813)
Sep 23 18:14:31 dist-test-slave-wsx5 krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187269, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.194@KRBTEST.COM
Sep 23 18:14:31 dist-test-slave-wsx5 krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187269, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.195@KRBTEST.COM
I20260923 18:14:31.206018 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:31.191884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:41813 (local address 127.27.75.193:38999)
0923 18:14:31.192028 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:31.192044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:31.192163 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:31.194037 (+ 1874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:31.194043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:31.194659 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:31.195257 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:31.195300 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:31.196669 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:31.196672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:31.197661 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:31.197673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:31.198411 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:31.198415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:31.198638 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:31.199356 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:31.199377 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:31.201181 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:31.203454 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:31.203461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:31.203477 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:31.203746 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:31.204154 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:31.204157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:31.204159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:31.204320 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:31.204993 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:31.205000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:31.205300 (+ 300us) client_negotiation.cc:769] Sending connection context
0923 18:14:31.205482 (+ 182us) client_negotiation.cc:240] Negotiation successful
0923 18:14:31.205818 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":88,"spinlock_wait_cycles":34560}
I20260923 18:14:31.206051 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:31.192083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:38999 (local address 127.27.75.194:41813)
0923 18:14:31.192396 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:31.192433 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:31.192475 (+ 42us) server_negotiation.cc:407] Connection header received
0923 18:14:31.193238 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:31.193245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:31.193630 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:31.193922 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:31.195464 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:31.196494 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:31.197794 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:31.198278 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:31.198448 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:31.201505 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:31.201515 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:31.201534 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:31.201572 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:31.203295 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:31.203868 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:31.203875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:31.203880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:31.204053 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:31.204413 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:31.204416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:31.204425 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:31.204867 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:31.205017 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:31.205614 (+ 597us) server_negotiation.cc:299] Negotiation successful
0923 18:14:31.205880 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":87,"threads_started":1}
I20260923 18:14:31.206511 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:31.191884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:41005 (local address 127.27.75.193:45479)
0923 18:14:31.192440 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:31.192454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:31.192558 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:31.194037 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:31.194043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:31.194659 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:31.195258 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:31.195276 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:31.196661 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:31.196664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:31.197660 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:31.197673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:31.198411 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:31.198414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:31.198636 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:31.199354 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:31.199373 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:31.201693 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:31.204439 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:31.204442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:31.204444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:31.204776 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:31.205165 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:31.205168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:31.205170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:31.205246 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:31.205974 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:31.205977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:31.206085 (+ 108us) client_negotiation.cc:769] Sending connection context
0923 18:14:31.206184 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 18:14:31.206330 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":260,"spinlock_wait_cycles":8832,"thread_start_us":64,"threads_started":1}
I20260923 18:14:31.207226 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:31.192158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:45479 (local address 127.27.75.195:41005)
0923 18:14:31.192522 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:31.192534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:31.192574 (+ 40us) server_negotiation.cc:407] Connection header received
0923 18:14:31.193282 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:31.193289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:31.193630 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:31.193922 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:31.195458 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:31.196493 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:31.197793 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:31.198278 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:31.198501 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:31.201908 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:31.201914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:31.201929 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:31.201977 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:31.204307 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:31.204908 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:31.204913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:31.204919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:31.205047 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:31.205332 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:31.205335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:31.205336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:31.205837 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:31.206000 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:31.206831 (+ 831us) server_negotiation.cc:299] Negotiation successful
0923 18:14:31.207075 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":94,"threads_started":1}
W20260923 18:14:31.208910 12058 leader_election.cc:341] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813): Not found: Tablet not found: e86bbe5c4b71422da5b40012b7c574a1
W20260923 18:14:31.209340 12061 leader_election.cc:341] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005): Not found: Tablet not found: e86bbe5c4b71422da5b40012b7c574a1
I20260923 18:14:31.209429 12061 leader_election.cc:302] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c163e090e5ac4b45a98da3fc9f72baec; no voters: 3b12bf3040754caf8e9269458ea1aa32, 6f433495bcd74994bf41c2b7ed29d3de
I20260923 18:14:31.209697 12448 raft_consensus.cc:2726] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:14:31.339061 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:31.338608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:58151)
0923 18:14:31.338892 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:31.338951 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":71,"threads_started":1}
I20260923 18:14:31.975530 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:31.975072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:38199)
0923 18:14:31.975342 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:31.975433 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":73,"threads_started":1}
I20260923 18:14:32.150609 12454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:32.150149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:44995)
0923 18:14:32.150438 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:32.150506 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":77,"threads_started":1}
I20260923 18:14:32.340240 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:32.339818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:42035)
0923 18:14:32.340066 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:32.340129 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260923 18:14:32.976810 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:32.976294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:37415)
0923 18:14:32.976648 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:32.976709 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":134,"threads_started":1}
I20260923 18:14:33.152087 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:33.151571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:42747)
0923 18:14:33.151879 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:33.151957 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
I20260923 18:14:33.286164 12458 raft_consensus.cc:482] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:33.286299 12458 raft_consensus.cc:504] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:33.286679 12458 leader_election.cc:288] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005), 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813)
W20260923 18:14:33.287655 12058 leader_election.cc:341] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813): Not found: Tablet not found: e86bbe5c4b71422da5b40012b7c574a1
W20260923 18:14:33.287781 12061 leader_election.cc:341] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005): Not found: Tablet not found: e86bbe5c4b71422da5b40012b7c574a1
I20260923 18:14:33.287864 12061 leader_election.cc:302] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c163e090e5ac4b45a98da3fc9f72baec; no voters: 3b12bf3040754caf8e9269458ea1aa32, 6f433495bcd74994bf41c2b7ed29d3de
I20260923 18:14:33.288019 12458 raft_consensus.cc:2726] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:14:33.341514 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:33.341047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:57931)
0923 18:14:33.341355 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:33.341414 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
I20260923 18:14:33.978238 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:33.977710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:59081)
0923 18:14:33.978065 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:33.978137 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":134,"threads_started":1}
I20260923 18:14:34.153441 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:34.152927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:33777)
0923 18:14:34.153272 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:34.153342 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
I20260923 18:14:34.342777 12462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:34.342319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:48729)
0923 18:14:34.342612 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:34.342680 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
I20260923 18:14:34.979628 12463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:34.979182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:54967)
0923 18:14:34.979467 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:34.979531 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
I20260923 18:14:34.983330 12464 raft_consensus.cc:482] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:34.983460 12464 raft_consensus.cc:504] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:34.983904 12464 leader_election.cc:288] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005), 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813)
W20260923 18:14:34.984740 12058 leader_election.cc:341] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813): Not found: Tablet not found: e86bbe5c4b71422da5b40012b7c574a1
W20260923 18:14:34.984876 12061 leader_election.cc:341] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005): Not found: Tablet not found: e86bbe5c4b71422da5b40012b7c574a1
I20260923 18:14:34.984967 12061 leader_election.cc:302] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c163e090e5ac4b45a98da3fc9f72baec; no voters: 3b12bf3040754caf8e9269458ea1aa32, 6f433495bcd74994bf41c2b7ed29d3de
I20260923 18:14:34.985147 12464 raft_consensus.cc:2726] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:14:35.154850 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:35.154391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:57281)
0923 18:14:35.154686 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:35.154752 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260923 18:14:35.344414 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:35.343749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:44699)
0923 18:14:35.344219 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:35.344282 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":93,"threads_started":1}
I20260923 18:14:35.980907 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:35.980455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:50129)
0923 18:14:35.980748 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:35.980811 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260923 18:14:36.156207 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:36.155713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:38885)
0923 18:14:36.156040 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:36.156109 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":93,"threads_started":1}
I20260923 18:14:36.345952 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:36.345497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:43113)
0923 18:14:36.345773 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:36.345839 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260923 18:14:36.982250 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:36.981811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:51047)
0923 18:14:36.982088 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:36.982153 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":73,"threads_started":1}
I20260923 18:14:37.157600 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:37.157063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:53795)
0923 18:14:37.157438 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:37.157505 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":100,"threads_started":1}
I20260923 18:14:37.347345 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:37.346784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:54901)
0923 18:14:37.347171 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:37.347240 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":101,"threads_started":1}
I20260923 18:14:37.983592 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:37.983112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:59483)
0923 18:14:37.983411 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:37.983491 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":79,"threads_started":1}
I20260923 18:14:38.158902 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:38.158375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:47217)
0923 18:14:38.158683 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:38.158800 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":90,"threads_started":1}
I20260923 18:14:38.348694 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:38.348201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:35469)
0923 18:14:38.348507 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:38.348594 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":90,"threads_started":1}
I20260923 18:14:38.985004 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:38.984477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:49343)
0923 18:14:38.984816 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:38.984900 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260923 18:14:39.153578 12477 raft_consensus.cc:482] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:39.153733 12477 raft_consensus.cc:504] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:39.154140 12477 leader_election.cc:288] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005), 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813)
W20260923 18:14:39.154969 12058 leader_election.cc:341] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813): Not found: Tablet not found: e86bbe5c4b71422da5b40012b7c574a1
W20260923 18:14:39.155088 12061 leader_election.cc:341] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005): Not found: Tablet not found: e86bbe5c4b71422da5b40012b7c574a1
I20260923 18:14:39.155182 12061 leader_election.cc:302] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c163e090e5ac4b45a98da3fc9f72baec; no voters: 3b12bf3040754caf8e9269458ea1aa32, 6f433495bcd74994bf41c2b7ed29d3de
I20260923 18:14:39.155333 12477 raft_consensus.cc:2726] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:14:39.160127 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:39.159691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:52619)
0923 18:14:39.159971 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:39.160033 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":103,"threads_started":1}
I20260923 18:14:39.350015 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:39.349556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:36685)
0923 18:14:39.349841 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:39.349910 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260923 18:14:39.986429 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:39.985990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:37993)
0923 18:14:39.986272 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:39.986334 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
I20260923 18:14:40.161514 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:40.161025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:58239)
0923 18:14:40.161340 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:40.161413 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":115,"threads_started":1}
I20260923 18:14:40.351286 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:40.350843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:58943)
0923 18:14:40.351104 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:40.351168 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260923 18:14:40.987721 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:40.987238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:48307)
0923 18:14:40.987553 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:40.987618 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":97,"threads_started":1}
I20260923 18:14:41.162783 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:41.162318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:50097)
0923 18:14:41.162612 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:41.162674 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
I20260923 18:14:41.352581 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:41.352120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:56967)
0923 18:14:41.352418 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:41.352484 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
I20260923 18:14:41.989238 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:41.988667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:41579)
0923 18:14:41.989048 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:41.989116 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":150,"threads_started":1}
I20260923 18:14:42.164163 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:42.163643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:42607)
0923 18:14:42.163987 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:42.164054 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":76,"threads_started":1}
I20260923 18:14:42.353977 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:42.353512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:56881)
0923 18:14:42.353811 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:42.353878 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":74,"threads_started":1}
I20260923 18:14:42.737524 12489 raft_consensus.cc:482] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:42.737700 12489 raft_consensus.cc:504] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12bf3040754caf8e9269458ea1aa32" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41005 } } peers { permanent_uuid: "6f433495bcd74994bf41c2b7ed29d3de" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 41813 } } peers { permanent_uuid: "c163e090e5ac4b45a98da3fc9f72baec" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 33403 } }
I20260923 18:14:42.738221 12489 leader_election.cc:288] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005), 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813)
W20260923 18:14:42.739117 12061 leader_election.cc:341] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b12bf3040754caf8e9269458ea1aa32 (127.27.75.195:41005): Not found: Tablet not found: e86bbe5c4b71422da5b40012b7c574a1
W20260923 18:14:42.739244 12058 leader_election.cc:341] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f433495bcd74994bf41c2b7ed29d3de (127.27.75.194:41813): Not found: Tablet not found: e86bbe5c4b71422da5b40012b7c574a1
I20260923 18:14:42.739315 12058 leader_election.cc:302] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c163e090e5ac4b45a98da3fc9f72baec; no voters: 3b12bf3040754caf8e9269458ea1aa32, 6f433495bcd74994bf41c2b7ed29d3de
I20260923 18:14:42.739457 12489 raft_consensus.cc:2726] T e86bbe5c4b71422da5b40012b7c574a1 P c163e090e5ac4b45a98da3fc9f72baec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 18:14:42.990607 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:42.990090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:43147)
0923 18:14:42.990407 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:42.990507 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":78,"threads_started":1}
I20260923 18:14:43.165637 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:43.165096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:34913)
0923 18:14:43.165459 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:43.165530 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
I20260923 18:14:43.355357 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:43.354932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:45383)
0923 18:14:43.355179 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:43.355248 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260923 18:14:43.991925 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:43.991450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:56871)
0923 18:14:43.991747 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:43.991825 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
I20260923 18:14:44.166954 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:44.166460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:43307)
0923 18:14:44.166748 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:44.166853 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
I20260923 18:14:44.356645 12495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:44.356158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.195:59083)
0923 18:14:44.356479 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:44.356544 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":81,"threads_started":1}
I20260923 18:14:44.993399 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:44.992929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.193:43191)
0923 18:14:44.993228 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:44.993296 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":67,"threads_started":1}
I20260923 18:14:45.168434 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:45.167954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42163 (local address 127.27.75.194:54899)
0923 18:14:45.168270 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:45.168338 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.254:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260923 18:14:45.347426 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 12007
I20260923 18:14:45.352674 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 12177
I20260923 18:14:45.357874 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 12312
2026-09-23T18:14:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59333 ms)
[----------] 30 tests from SecurityITest (242863 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:14:45 dist-test-slave-wsx5 krb5kdc[12501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:14:45 dist-test-slave-wsx5 krb5kdc[12501](info): set up 2 sockets
Sep 23 18:14:45 dist-test-slave-wsx5 krb5kdc[12501](info): commencing operation
krb5kdc: starting...
W20260923 18:14:46.319130 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:14:46 dist-test-slave-wsx5 krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:14:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:14:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:14:46.446545 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:35203
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:35203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:46.564751 12514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:46.565040 12514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:46.565083 12514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:46.569300 12514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:14:46.569378 12514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:46.569396 12514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:46.569412 12514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:14:46.569427 12514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:14:46.575121 12514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:35203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:35203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:46.576201 12514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:46.577245 12514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:46.586814 12523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:46.586894 12521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:46.586863 12520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:46.587181 12514 server_base.cc:1034] running on GCE node
I20260923 18:14:46.587766 12514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:46.588989 12514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:46.590160 12514 hybrid_clock.cc:648] HybridClock initialized: now 1790187286590134 us; error 46 us; skew 500 ppm
Sep 23 18:14:46 dist-test-slave-wsx5 krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187286, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:46.593000 12514 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:14:46.594053 12514 webserver.cc:460] Webserver started at http://127.27.75.254:39131/ using document root <none> and password file <none>
I20260923 18:14:46.594650 12514 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:46.594725 12514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:46.594931 12514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:46.597080 12514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "14fcddd7d32546a9a7431daa3608e40a"
format_stamp: "Formatted at 2026-09-23 18:14:46 on dist-test-slave-wsx5"
server_key: "f5a024cdd739df1206781e88202fbe27"
server_key_iv: "94fe337c5fd6104c886dcb2f9c44569d"
server_key_version: "encryptionkey@0"
I20260923 18:14:46.597692 12514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "14fcddd7d32546a9a7431daa3608e40a"
format_stamp: "Formatted at 2026-09-23 18:14:46 on dist-test-slave-wsx5"
server_key: "f5a024cdd739df1206781e88202fbe27"
server_key_iv: "94fe337c5fd6104c886dcb2f9c44569d"
server_key_version: "encryptionkey@0"
I20260923 18:14:46.601749 12514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:14:46.604341 12530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:46.605859 12514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:46.606011 12514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "14fcddd7d32546a9a7431daa3608e40a"
format_stamp: "Formatted at 2026-09-23 18:14:46 on dist-test-slave-wsx5"
server_key: "f5a024cdd739df1206781e88202fbe27"
server_key_iv: "94fe337c5fd6104c886dcb2f9c44569d"
server_key_version: "encryptionkey@0"
I20260923 18:14:46.606133 12514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:46.630602 12514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:46.634079 12514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:46.634284 12514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:46.641209 12514 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:35203
I20260923 18:14:46.641243 12582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:35203 every 8 connection(s)
I20260923 18:14:46.642161 12514 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:14:46.643656 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 12514
I20260923 18:14:46.643784 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:14:46.644099 27951 external_mini_cluster.cc:1428] Setting key df8a0ee7fd13f5382c5234a20a05940d
I20260923 18:14:46.645397 12583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:46.651890 12583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a: Bootstrap starting.
Sep 23 18:14:46 dist-test-slave-wsx5 krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:46.654515 12583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:46.655290 12583 log.cc:824] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:46.657456 12583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a: No bootstrap required, opened a new log
I20260923 18:14:46.658540 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:46.645482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.27.75.254:35203)
0923 18:14:46.645907 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:46.645923 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:46.645961 (+ 38us) server_negotiation.cc:407] Connection header received
0923 18:14:46.646614 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:46.646632 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:46.646970 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:46.647248 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:46.648180 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:46.649392 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:46.650581 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:46.651048 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:46.651165 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:46.653906 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:46.653911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:46.653925 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:46.653957 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:46.655664 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:46.656082 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:46.656087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:46.656092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:46.656202 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:46.656514 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:46.656516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:46.656518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:46.656899 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:46.657051 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:46.657447 (+ 396us) server_negotiation.cc:299] Negotiation successful
0923 18:14:46.657754 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":119,"threads_started":1}
I20260923 18:14:46.660560 12583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fcddd7d32546a9a7431daa3608e40a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35203 } }
I20260923 18:14:46.660802 12583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:46.660878 12583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14fcddd7d32546a9a7431daa3608e40a, State: Initialized, Role: FOLLOWER
I20260923 18:14:46.661373 12583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fcddd7d32546a9a7431daa3608e40a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35203 } }
I20260923 18:14:46.661504 12583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:14:46.661559 12583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:14:46.661654 12583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:46.662684 12583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fcddd7d32546a9a7431daa3608e40a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35203 } }
I20260923 18:14:46.663049 12583 leader_election.cc:302] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14fcddd7d32546a9a7431daa3608e40a; no voters:
I20260923 18:14:46.663355 12583 leader_election.cc:288] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:14:46.663511 12588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:46.663789 12588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [term 1 LEADER]: Becoming Leader. State: Replica: 14fcddd7d32546a9a7431daa3608e40a, State: Running, Role: LEADER
I20260923 18:14:46.664080 12588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fcddd7d32546a9a7431daa3608e40a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35203 } }
I20260923 18:14:46.664542 12583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:14:46.665643 12590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14fcddd7d32546a9a7431daa3608e40a. Latest consensus state: current_term: 1 leader_uuid: "14fcddd7d32546a9a7431daa3608e40a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fcddd7d32546a9a7431daa3608e40a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35203 } } }
I20260923 18:14:46.665773 12590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:46.666136 12597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:14:46.666489 12589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14fcddd7d32546a9a7431daa3608e40a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fcddd7d32546a9a7431daa3608e40a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35203 } } }
I20260923 18:14:46.666620 12589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:46.669100 12597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:14:46.674695 12597 catalog_manager.cc:1347] Generated new cluster ID: 06ecb4e5925e4f1e8570e4c06a37645a
I20260923 18:14:46.674777 12597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:14:46.691500 12597 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:14:46.692514 12597 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:14:46.698185 12597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a: Generated new TSK 0
I20260923 18:14:46.698922 12597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:14:46.737008 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35203
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:46.850510 12609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:46.850809 12609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:46.850873 12609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:46.855129 12609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:46.855204 12609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:46.855291 12609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:14:46.860898 12609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:46.862109 12609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:46.863154 12609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:14:46.873503 12609 server_base.cc:1034] running on GCE node
W20260923 18:14:46.873410 12617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:46.873395 12615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:46.873397 12614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:46.874106 12609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:46.874802 12609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:46.875965 12609 hybrid_clock.cc:648] HybridClock initialized: now 1790187286875931 us; error 48 us; skew 500 ppm
Sep 23 18:14:46 dist-test-slave-wsx5 krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187286, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:46.878823 12609 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:14:46.879828 12609 webserver.cc:460] Webserver started at http://127.27.75.193:41545/ using document root <none> and password file <none>
I20260923 18:14:46.880409 12609 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:46.880482 12609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:46.880699 12609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:46.882845 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "8641942a4e3741ba9a5b47dfc93105a4"
format_stamp: "Formatted at 2026-09-23 18:14:46 on dist-test-slave-wsx5"
server_key: "a4e90c94b834b08ff9f48087e095f573"
server_key_iv: "490528d0f76c8cf8fd7e65091cc0879a"
server_key_version: "encryptionkey@0"
I20260923 18:14:46.883400 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "8641942a4e3741ba9a5b47dfc93105a4"
format_stamp: "Formatted at 2026-09-23 18:14:46 on dist-test-slave-wsx5"
server_key: "a4e90c94b834b08ff9f48087e095f573"
server_key_iv: "490528d0f76c8cf8fd7e65091cc0879a"
server_key_version: "encryptionkey@0"
I20260923 18:14:46.887280 12609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 18:14:46.889865 12624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:46.891353 12609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:46.891484 12609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "8641942a4e3741ba9a5b47dfc93105a4"
format_stamp: "Formatted at 2026-09-23 18:14:46 on dist-test-slave-wsx5"
server_key: "a4e90c94b834b08ff9f48087e095f573"
server_key_iv: "490528d0f76c8cf8fd7e65091cc0879a"
server_key_version: "encryptionkey@0"
I20260923 18:14:46.891623 12609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:46.911118 12609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:46.914682 12609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:46.914866 12609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:46.915463 12609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:46.916298 12609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:46.916347 12609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:46.916399 12609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:46.916427 12609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:46.925410 12609 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:38973
I20260923 18:14:46.925446 12738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:38973 every 8 connection(s)
I20260923 18:14:46.926343 12609 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:14:46.934132 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 12609
I20260923 18:14:46.934254 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:14:46.934540 27951 external_mini_cluster.cc:1428] Setting key 8ec326be921e9aa5d3deaaadcabfdf59
Sep 23 18:14:46 dist-test-slave-wsx5 krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187286, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:46.941469 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:46.928285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:46695 (local address 127.27.75.254:35203)
0923 18:14:46.928475 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:46.928479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:46.929485 (+ 1006us) server_negotiation.cc:407] Connection header received
0923 18:14:46.930865 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:46.930869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:46.930932 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:46.931007 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:46.932222 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:46.932781 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:46.934091 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:46.934418 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:46.934524 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:46.937323 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:46.937327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:46.937329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:46.937355 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:46.938922 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:46.939552 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:46.939555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:46.939557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:46.939644 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:46.940059 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:46.940062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:46.940063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:46.940267 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:46.940356 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:46.941096 (+ 740us) server_negotiation.cc:299] Negotiation successful
0923 18:14:46.941296 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 18:14:46.942324 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:46.928694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:35203 (local address 127.27.75.193:46695)
0923 18:14:46.929245 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:46.929359 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:46.930630 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:46.931198 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:46.931206 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:46.931569 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:46.932072 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:46.932085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:46.932909 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:46.932912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:46.933942 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:46.933955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:46.934601 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:46.934606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:46.934831 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:46.935454 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:46.935471 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:46.937108 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:46.939061 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:46.939067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:46.939083 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:46.939416 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:46.939737 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:46.939741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:46.939743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:46.939949 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:46.940370 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:46.940378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:46.940753 (+ 375us) client_negotiation.cc:769] Sending connection context
0923 18:14:46.940989 (+ 236us) client_negotiation.cc:240] Negotiation successful
0923 18:14:46.941402 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":118,"threads_started":1}
I20260923 18:14:46.943744 12739 heartbeater.cc:347] Connected to a master server at 127.27.75.254:35203
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:14:46.944054 12739 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:46.944602 12739 heartbeater.cc:511] Master 127.27.75.254:35203 requested a full tablet report, sending...
I20260923 18:14:46.946442 12547 ts_manager.cc:194] Registered new tserver with Master: 8641942a4e3741ba9a5b47dfc93105a4 (127.27.75.193:38973)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:14:46.947793 12547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:46695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:14:46.964795 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35203
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:47.077087 12744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:47.077432 12744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:47.077477 12744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:47.082047 12744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:47.082126 12744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:47.082193 12744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:14:47.087962 12744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:47.089658 12744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:47.091159 12744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:14:47.101178 12744 server_base.cc:1034] running on GCE node
W20260923 18:14:47.101135 12749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:47.101289 12750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:47.101091 12752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:47.101826 12744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:47.102480 12744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:47.103634 12744 hybrid_clock.cc:648] HybridClock initialized: now 1790187287103616 us; error 31 us; skew 500 ppm
Sep 23 18:14:47 dist-test-slave-wsx5 krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187287, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:47.106443 12744 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:14:47.107339 12744 webserver.cc:460] Webserver started at http://127.27.75.194:36167/ using document root <none> and password file <none>
I20260923 18:14:47.107872 12744 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:47.107933 12744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:47.108139 12744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:47.110409 12744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3"
format_stamp: "Formatted at 2026-09-23 18:14:47 on dist-test-slave-wsx5"
server_key: "bd9d07f69a570d4c8305507ab8460c93"
server_key_iv: "6d0b877736d1c763240c709a7154f729"
server_key_version: "encryptionkey@0"
I20260923 18:14:47.111023 12744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3"
format_stamp: "Formatted at 2026-09-23 18:14:47 on dist-test-slave-wsx5"
server_key: "bd9d07f69a570d4c8305507ab8460c93"
server_key_iv: "6d0b877736d1c763240c709a7154f729"
server_key_version: "encryptionkey@0"
I20260923 18:14:47.115005 12744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:14:47.117658 12759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:47.119071 12744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:47.119202 12744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3"
format_stamp: "Formatted at 2026-09-23 18:14:47 on dist-test-slave-wsx5"
server_key: "bd9d07f69a570d4c8305507ab8460c93"
server_key_iv: "6d0b877736d1c763240c709a7154f729"
server_key_version: "encryptionkey@0"
I20260923 18:14:47.119313 12744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:47.132030 12744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:47.135375 12744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:47.135563 12744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:47.136343 12744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:47.137353 12744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:47.137406 12744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:47.137465 12744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:47.137491 12744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:47.146034 12744 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:46825
I20260923 18:14:47.146091 12873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:46825 every 8 connection(s)
I20260923 18:14:47.146929 12744 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:14:47.151661 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 12744
I20260923 18:14:47.151814 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:14:47.152137 27951 external_mini_cluster.cc:1428] Setting key 97b72ddcb07d2766a92f7a50926c26b9
Sep 23 18:14:47 dist-test-slave-wsx5 krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187287, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:14:47.162567 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.148787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:49419 (local address 127.27.75.254:35203)
0923 18:14:47.148983 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:47.148988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:47.149770 (+ 782us) server_negotiation.cc:407] Connection header received
0923 18:14:47.151136 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:47.151139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:47.151193 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:47.151272 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:47.153020 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.153773 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.155444 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.155703 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.155810 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.158503 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:47.158507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:47.158510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:47.158543 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:47.159952 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:47.160648 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:47.160651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:47.160654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:47.160746 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:47.161224 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:47.161227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:47.161228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:47.161459 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:47.161547 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:47.162312 (+ 765us) server_negotiation.cc:299] Negotiation successful
0923 18:14:47.162440 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 18:14:47.163437 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.149156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:35203 (local address 127.27.75.194:49419)
0923 18:14:47.149619 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:47.149664 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:47.150896 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:47.151543 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:47.151553 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:47.152128 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:47.152817 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.152839 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.153923 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.153927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.155275 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.155287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.155881 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.155885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.156070 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.156684 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:47.156701 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:47.158303 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:47.160089 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:47.160097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:47.160118 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:47.160514 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:47.160855 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:47.160859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:47.160861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:47.161081 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:47.161585 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:47.161593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:47.161980 (+ 387us) client_negotiation.cc:769] Sending connection context
0923 18:14:47.162220 (+ 240us) client_negotiation.cc:240] Negotiation successful
0923 18:14:47.162561 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
I20260923 18:14:47.165019 12874 heartbeater.cc:347] Connected to a master server at 127.27.75.254:35203
I20260923 18:14:47.165298 12874 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:47.165853 12874 heartbeater.cc:511] Master 127.27.75.254:35203 requested a full tablet report, sending...
I20260923 18:14:47.166980 12547 ts_manager.cc:194] Registered new tserver with Master: b7cb25ab6b044a6e8b1bc756a8f625c3 (127.27.75.194:46825)
I20260923 18:14:47.167586 12547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:49419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:14:47.180634 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35203
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:47.294210 12879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:47.294505 12879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:47.294569 12879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:47.298735 12879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:47.298806 12879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:47.298895 12879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:14:47.304435 12879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:35203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:47.305626 12879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:47.306679 12879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:47.316725 12884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:47.316699 12885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:47.316844 12879 server_base.cc:1034] running on GCE node
W20260923 18:14:47.316725 12887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:47.317482 12879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:47.318122 12879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:47.319288 12879 hybrid_clock.cc:648] HybridClock initialized: now 1790187287319268 us; error 34 us; skew 500 ppm
Sep 23 18:14:47 dist-test-slave-wsx5 krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187287, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:47.322069 12879 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:14:47.323067 12879 webserver.cc:460] Webserver started at http://127.27.75.195:33387/ using document root <none> and password file <none>
I20260923 18:14:47.323657 12879 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:47.323724 12879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:47.323966 12879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:47.326112 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "824f1130c51c4d6fb0f05b5dc315bc30"
format_stamp: "Formatted at 2026-09-23 18:14:47 on dist-test-slave-wsx5"
server_key: "c6d4b5761a73f72543d94d7aeb2c3b1e"
server_key_iv: "ef28633776121c4afbae4ac47b23dfd0"
server_key_version: "encryptionkey@0"
I20260923 18:14:47.326680 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "824f1130c51c4d6fb0f05b5dc315bc30"
format_stamp: "Formatted at 2026-09-23 18:14:47 on dist-test-slave-wsx5"
server_key: "c6d4b5761a73f72543d94d7aeb2c3b1e"
server_key_iv: "ef28633776121c4afbae4ac47b23dfd0"
server_key_version: "encryptionkey@0"
I20260923 18:14:47.330603 12879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:14:47.333235 12894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:47.334769 12879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 18:14:47.334887 12879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "824f1130c51c4d6fb0f05b5dc315bc30"
format_stamp: "Formatted at 2026-09-23 18:14:47 on dist-test-slave-wsx5"
server_key: "c6d4b5761a73f72543d94d7aeb2c3b1e"
server_key_iv: "ef28633776121c4afbae4ac47b23dfd0"
server_key_version: "encryptionkey@0"
I20260923 18:14:47.334985 12879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:47.358857 12879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:47.362186 12879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:47.362349 12879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:47.362926 12879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:47.363775 12879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:47.363824 12879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:47.363865 12879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:47.363886 12879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:47.372603 12879 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:45565
I20260923 18:14:47.372747 13007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:45565 every 8 connection(s)
I20260923 18:14:47.373668 12879 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:14:47.377477 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 12879
I20260923 18:14:47.377612 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:14:47.377871 27951 external_mini_cluster.cc:1428] Setting key ecfe9f5c3059dd0f69f36750c1061134
Sep 23 18:14:47 dist-test-slave-wsx5 krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187287, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:47.389035 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.375743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:58979 (local address 127.27.75.254:35203)
0923 18:14:47.375931 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:47.375936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:47.376798 (+ 862us) server_negotiation.cc:407] Connection header received
0923 18:14:47.378188 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:47.378192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:47.378251 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:47.378337 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:47.379976 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.380697 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.382031 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.382264 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.382358 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.385323 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:47.385327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:47.385329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:47.385361 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:47.386810 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:47.387405 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:47.387408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:47.387409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:47.387500 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:47.387859 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:47.387862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:47.387863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:47.388077 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:47.388165 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:47.388721 (+ 556us) server_negotiation.cc:299] Negotiation successful
0923 18:14:47.388860 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 18:14:47.389961 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.376137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:35203 (local address 127.27.75.195:58979)
0923 18:14:47.376632 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:47.376681 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:47.377975 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:47.378551 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:47.378562 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:47.379078 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:47.379828 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.379847 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.380829 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.380833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.381897 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.381908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.382417 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.382422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.382605 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.383331 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:47.383368 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:47.385080 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:47.386955 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:47.386963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:47.386984 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:47.387275 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:47.387595 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:47.387598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:47.387601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:47.387759 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:47.388180 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:47.388188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:47.388476 (+ 288us) client_negotiation.cc:769] Sending connection context
0923 18:14:47.388647 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 18:14:47.389010 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":113,"threads_started":1}
I20260923 18:14:47.391206 13008 heartbeater.cc:347] Connected to a master server at 127.27.75.254:35203
I20260923 18:14:47.391491 13008 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:47.392284 13008 heartbeater.cc:511] Master 127.27.75.254:35203 requested a full tablet report, sending...
I20260923 18:14:47.393527 12547 ts_manager.cc:194] Registered new tserver with Master: 824f1130c51c4d6fb0f05b5dc315bc30 (127.27.75.195:45565)
I20260923 18:14:47.394105 12547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:58979
I20260923 18:14:47.404750 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:14:47.414575 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.407026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.27.75.254:35203)
0923 18:14:47.407183 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:47.407188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:47.407224 (+ 36us) server_negotiation.cc:407] Connection header received
0923 18:14:47.407381 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:47.407385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:47.407433 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:47.407501 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:47.408300 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.408866 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.409773 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.410004 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.410095 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.411430 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:47.411434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:47.411438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:47.411464 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:47.412859 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:47.413323 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:47.413326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:47.413328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:47.413417 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:47.413664 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:47.413668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:47.413670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:47.413896 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:47.414019 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:47.414303 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 18:14:47.414417 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 18:14:47.417460 12547 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:14:47.420593 12547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:14:47.435879 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.429948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.27.75.195:45565)
0923 18:14:47.430610 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:47.430623 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:47.430647 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:14:47.430724 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:47.430730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:47.430950 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:47.431099 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:47.431585 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.432758 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.434432 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.435127 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.435232 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.435392 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:47.435605 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 18:14:47.435716 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":154,"threads_started":1}
I20260923 18:14:47.436002 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.429979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:38973 (local address 127.0.0.1:59822)
0923 18:14:47.430488 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:47.430501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:47.430595 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:47.431676 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:47.431679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:47.431697 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:47.431801 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.431808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.433397 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.433401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.434569 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.434578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.435623 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.435626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.435708 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.435728 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 18:14:47.435785 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 18:14:47.435843 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":2,"thread_start_us":122,"threads_started":1}
I20260923 18:14:47.435889 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.430509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:46825 (local address 127.0.0.1:51184)
0923 18:14:47.430967 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:47.430980 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:47.431061 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:47.431919 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:47.431922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:47.431940 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:47.432041 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.432047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.433026 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.433030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.434223 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.434233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.435357 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.435361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.435485 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.435513 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 18:14:47.435560 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 18:14:47.435722 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260923 18:14:47.436398 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.430907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.27.75.194:46825)
0923 18:14:47.431503 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:47.431515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:47.431537 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:14:47.431609 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:47.431614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:47.431783 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:47.431925 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:47.432140 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.432895 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.434395 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.435196 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.435960 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.436104 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:47.436155 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 18:14:47.436245 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":142,"threads_started":1}
I20260923 18:14:47.436398 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.430514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.27.75.193:38973)
0923 18:14:47.431272 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:47.431287 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:47.431307 (+ 20us) server_negotiation.cc:407] Connection header received
0923 18:14:47.431374 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:47.431379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:47.431556 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:47.431700 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:47.432170 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.433267 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.434685 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.435494 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.435960 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.436103 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:47.436155 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 18:14:47.436258 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":86,"threads_started":1}
I20260923 18:14:47.435881 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.429768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:45565 (local address 127.0.0.1:43188)
0923 18:14:47.430110 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:47.430163 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:47.430295 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:47.431288 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:47.431295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:47.431318 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:47.431456 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.431462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.432906 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.432910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.434287 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.434299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.435254 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.435257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.435407 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.435451 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 18:14:47.435560 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 18:14:47.435728 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":72,"thread_start_us":73,"threads_started":1}
I20260923 18:14:47.439428 12942 tablet_service.cc:1467] Processing CreateTablet for tablet fac9fe26bb1b4e709c53b36b89bb8366 (DEFAULT_TABLE table=test-table [id=331f1bc9882e4e02be8ed2d9d9ede1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:47.439883 12673 tablet_service.cc:1467] Processing CreateTablet for tablet fac9fe26bb1b4e709c53b36b89bb8366 (DEFAULT_TABLE table=test-table [id=331f1bc9882e4e02be8ed2d9d9ede1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:47.440692 12942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac9fe26bb1b4e709c53b36b89bb8366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:47.440789 12673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac9fe26bb1b4e709c53b36b89bb8366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:47.447594 13026 tablet_bootstrap.cc:492] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4: Bootstrap starting.
I20260923 18:14:47.447721 12808 tablet_service.cc:1467] Processing CreateTablet for tablet fac9fe26bb1b4e709c53b36b89bb8366 (DEFAULT_TABLE table=test-table [id=331f1bc9882e4e02be8ed2d9d9ede1c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:47.448824 12808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac9fe26bb1b4e709c53b36b89bb8366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:47.449573 13027 tablet_bootstrap.cc:492] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30: Bootstrap starting.
I20260923 18:14:47.450352 13026 tablet_bootstrap.cc:654] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:47.451263 13026 log.cc:824] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:47.451826 13027 tablet_bootstrap.cc:654] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:47.453176 13027 log.cc:824] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:47.453858 13026 tablet_bootstrap.cc:492] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4: No bootstrap required, opened a new log
I20260923 18:14:47.454075 13026 ts_tablet_manager.cc:1397] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:14:47.455304 13027 tablet_bootstrap.cc:492] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30: No bootstrap required, opened a new log
I20260923 18:14:47.455520 13027 ts_tablet_manager.cc:1397] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:14:47.455564 13030 tablet_bootstrap.cc:492] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3: Bootstrap starting.
I20260923 18:14:47.457858 13030 tablet_bootstrap.cc:654] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:47.458588 13026 raft_consensus.cc:348] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 45565 } } peers { permanent_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46825 } } peers { permanent_uuid: "8641942a4e3741ba9a5b47dfc93105a4" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38973 } }
I20260923 18:14:47.458904 13026 raft_consensus.cc:374] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:47.458950 13026 raft_consensus.cc:729] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8641942a4e3741ba9a5b47dfc93105a4, State: Initialized, Role: FOLLOWER
I20260923 18:14:47.459017 13030 log.cc:824] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:47.459414 13027 raft_consensus.cc:348] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 45565 } } peers { permanent_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46825 } } peers { permanent_uuid: "8641942a4e3741ba9a5b47dfc93105a4" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38973 } }
I20260923 18:14:47.459687 13027 raft_consensus.cc:374] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:47.459751 13027 raft_consensus.cc:729] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824f1130c51c4d6fb0f05b5dc315bc30, State: Initialized, Role: FOLLOWER
I20260923 18:14:47.459887 13026 consensus_queue.cc:260] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 45565 } } peers { permanent_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46825 } } peers { permanent_uuid: "8641942a4e3741ba9a5b47dfc93105a4" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38973 } }
I20260923 18:14:47.460180 13027 consensus_queue.cc:260] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 45565 } } peers { permanent_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46825 } } peers { permanent_uuid: "8641942a4e3741ba9a5b47dfc93105a4" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38973 } }
I20260923 18:14:47.460857 13008 heartbeater.cc:503] Master 127.27.75.254:35203 was elected leader, sending a full tablet report...
I20260923 18:14:47.461086 13027 ts_tablet_manager.cc:1428] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 18:14:47.461086 13026 ts_tablet_manager.cc:1428] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260923 18:14:47.461438 12739 heartbeater.cc:503] Master 127.27.75.254:35203 was elected leader, sending a full tablet report...
I20260923 18:14:47.461665 13030 tablet_bootstrap.cc:492] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3: No bootstrap required, opened a new log
I20260923 18:14:47.461889 13030 ts_tablet_manager.cc:1397] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:14:47.466054 13030 raft_consensus.cc:348] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 45565 } } peers { permanent_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46825 } } peers { permanent_uuid: "8641942a4e3741ba9a5b47dfc93105a4" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38973 } }
I20260923 18:14:47.466395 13030 raft_consensus.cc:374] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:47.466461 13030 raft_consensus.cc:729] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7cb25ab6b044a6e8b1bc756a8f625c3, State: Initialized, Role: FOLLOWER
I20260923 18:14:47.466954 13030 consensus_queue.cc:260] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 45565 } } peers { permanent_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46825 } } peers { permanent_uuid: "8641942a4e3741ba9a5b47dfc93105a4" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38973 } }
I20260923 18:14:47.467593 12874 heartbeater.cc:503] Master 127.27.75.254:35203 was elected leader, sending a full tablet report...
I20260923 18:14:47.467947 13030 ts_tablet_manager.cc:1428] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260923 18:14:47.553999 13032 raft_consensus.cc:482] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:47.554203 13032 raft_consensus.cc:504] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 45565 } } peers { permanent_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46825 } } peers { permanent_uuid: "8641942a4e3741ba9a5b47dfc93105a4" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38973 } }
I20260923 18:14:47.555312 13032 leader_election.cc:288] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 824f1130c51c4d6fb0f05b5dc315bc30 (127.27.75.195:45565), b7cb25ab6b044a6e8b1bc756a8f625c3 (127.27.75.194:46825)
I20260923 18:14:47.559718 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.555681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:53439 (local address 127.27.75.195:45565)
0923 18:14:47.555876 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:47.555881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:47.556066 (+ 185us) server_negotiation.cc:407] Connection header received
0923 18:14:47.556300 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:47.556304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:47.556361 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:47.556434 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:47.556758 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.557376 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.558671 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.559102 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.559224 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.559280 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:47.559494 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 18:14:47.559581 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 18:14:47.559748 13035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.555592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:45565 (local address 127.27.75.193:53439)
0923 18:14:47.555971 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:47.555985 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:47.556148 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:47.556458 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:47.556461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:47.556480 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:47.556623 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.556631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.557484 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.557488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.558528 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.558538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.559243 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.559247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.559352 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.559385 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 18:14:47.559457 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:14:47.559548 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":22,"thread_start_us":134,"threads_started":1}
I20260923 18:14:47.559870 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.556155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:57881 (local address 127.27.75.194:46825)
0923 18:14:47.556290 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:47.556295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:47.556387 (+ 92us) server_negotiation.cc:407] Connection header received
0923 18:14:47.556522 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:47.556524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:47.556572 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:47.556656 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:47.557006 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.557600 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.558778 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.559318 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.559463 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.559512 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:47.559674 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 18:14:47.559762 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:14:47.560000 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.555994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:46825 (local address 127.27.75.193:57881)
0923 18:14:47.556318 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:47.556330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:47.556426 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:47.556728 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:47.556732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:47.556752 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:47.556867 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.556873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.557715 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.557718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.558584 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:47.558595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.559446 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.559450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:47.559547 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.559570 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:14:47.559647 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 18:14:47.559729 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":36,"thread_start_us":88,"threads_started":1}
I20260923 18:14:47.560526 12962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac9fe26bb1b4e709c53b36b89bb8366" candidate_uuid: "8641942a4e3741ba9a5b47dfc93105a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" is_pre_election: true
I20260923 18:14:47.560715 12828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac9fe26bb1b4e709c53b36b89bb8366" candidate_uuid: "8641942a4e3741ba9a5b47dfc93105a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" is_pre_election: true
I20260923 18:14:47.560896 12962 raft_consensus.cc:2454] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8641942a4e3741ba9a5b47dfc93105a4 in term 0.
I20260923 18:14:47.560957 12828 raft_consensus.cc:2454] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8641942a4e3741ba9a5b47dfc93105a4 in term 0.
I20260923 18:14:47.561431 12627 leader_election.cc:302] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 824f1130c51c4d6fb0f05b5dc315bc30, 8641942a4e3741ba9a5b47dfc93105a4; no voters:
I20260923 18:14:47.561692 13032 raft_consensus.cc:2781] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:14:47.561793 13032 raft_consensus.cc:482] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:14:47.561842 13032 raft_consensus.cc:3037] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:47.562883 13032 raft_consensus.cc:504] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 45565 } } peers { permanent_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46825 } } peers { permanent_uuid: "8641942a4e3741ba9a5b47dfc93105a4" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38973 } }
I20260923 18:14:47.563232 13032 leader_election.cc:288] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [CANDIDATE]: Term 1 election: Requested vote from peers 824f1130c51c4d6fb0f05b5dc315bc30 (127.27.75.195:45565), b7cb25ab6b044a6e8b1bc756a8f625c3 (127.27.75.194:46825)
I20260923 18:14:47.563638 12962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac9fe26bb1b4e709c53b36b89bb8366" candidate_uuid: "8641942a4e3741ba9a5b47dfc93105a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "824f1130c51c4d6fb0f05b5dc315bc30"
I20260923 18:14:47.563728 12828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac9fe26bb1b4e709c53b36b89bb8366" candidate_uuid: "8641942a4e3741ba9a5b47dfc93105a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3"
I20260923 18:14:47.563838 12962 raft_consensus.cc:3037] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:47.563876 12828 raft_consensus.cc:3037] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:47.565100 12962 raft_consensus.cc:2454] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8641942a4e3741ba9a5b47dfc93105a4 in term 1.
I20260923 18:14:47.565104 12828 raft_consensus.cc:2454] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8641942a4e3741ba9a5b47dfc93105a4 in term 1.
I20260923 18:14:47.565491 12627 leader_election.cc:302] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 824f1130c51c4d6fb0f05b5dc315bc30, 8641942a4e3741ba9a5b47dfc93105a4; no voters:
I20260923 18:14:47.565671 13032 raft_consensus.cc:2781] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:47.565938 13032 raft_consensus.cc:686] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [term 1 LEADER]: Becoming Leader. State: Replica: 8641942a4e3741ba9a5b47dfc93105a4, State: Running, Role: LEADER
I20260923 18:14:47.566337 13032 consensus_queue.cc:237] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 45565 } } peers { permanent_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46825 } } peers { permanent_uuid: "8641942a4e3741ba9a5b47dfc93105a4" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38973 } }
I20260923 18:14:47.569665 12546 catalog_manager.cc:5495] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8641942a4e3741ba9a5b47dfc93105a4 (127.27.75.193). New cstate: current_term: 1 leader_uuid: "8641942a4e3741ba9a5b47dfc93105a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 45565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8641942a4e3741ba9a5b47dfc93105a4" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 38973 } health_report { overall_health: HEALTHY } } }
I20260923 18:14:47.592240 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.588148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.27.75.193:38973)
0923 18:14:47.588280 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:47.588283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:47.588297 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:14:47.588461 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:47.588463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:47.588533 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:47.588632 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:14:47.588968 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.589527 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.590547 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.590827 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.590921 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.591364 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:14:47.591817 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:14:47.591917 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:47.592054 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 18:14:47.592120 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 18:14:47.598618 12828 raft_consensus.cc:1261] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3 [term 1 FOLLOWER]: Refusing update from remote peer 8641942a4e3741ba9a5b47dfc93105a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:47.598618 12962 raft_consensus.cc:1261] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30 [term 1 FOLLOWER]: Refusing update from remote peer 8641942a4e3741ba9a5b47dfc93105a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:47.599323 13037 consensus_queue.cc:1035] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "824f1130c51c4d6fb0f05b5dc315bc30" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:47.599515 13032 consensus_queue.cc:1035] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7cb25ab6b044a6e8b1bc756a8f625c3" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:47.615715 12546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:47.615897 12546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:47.618599 12546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14fcddd7d32546a9a7431daa3608e40a: Sending DeleteTablet for 3 replicas of tablet fac9fe26bb1b4e709c53b36b89bb8366
I20260923 18:14:47.619560 12673 tablet_service.cc:1514] Processing DeleteTablet for tablet fac9fe26bb1b4e709c53b36b89bb8366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:47 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:59822
I20260923 18:14:47.619572 12808 tablet_service.cc:1514] Processing DeleteTablet for tablet fac9fe26bb1b4e709c53b36b89bb8366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:47 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:51184
I20260923 18:14:47.619764 12942 tablet_service.cc:1514] Processing DeleteTablet for tablet fac9fe26bb1b4e709c53b36b89bb8366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:47 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:43188
I20260923 18:14:47.620328 13051 tablet_replica.cc:330] T fac9fe26bb1b4e709c53b36b89bb8366 P 8641942a4e3741ba9a5b47dfc93105a4: stopping tablet replica
I20260923 18:14:47.620332 13050 tablet_replica.cc:330] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3: stopping tablet replica
I20260923 18:14:47.620558 13052 tablet_replica.cc:330] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30: stopping tablet replica
I20260923 18:14:47.620728 13050 raft_consensus.cc:2229] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:47.620757 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 12609
I20260923 18:14:47.622534 13052 raft_consensus.cc:2229] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:47.622823 13052 raft_consensus.cc:2258] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:47.622833 13050 raft_consensus.cc:2258] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:47.624441 13050 ts_tablet_manager.cc:1905] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 18:14:47.626822 12534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:38973 (error 108)
I20260923 18:14:47.626940 13050 ts_tablet_manager.cc:1918] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:47.627069 13050 log.cc:1196] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/fac9fe26bb1b4e709c53b36b89bb8366
I20260923 18:14:47.627426 13050 ts_tablet_manager.cc:1939] T fac9fe26bb1b4e709c53b36b89bb8366 P b7cb25ab6b044a6e8b1bc756a8f625c3: Deleting consensus metadata
I20260923 18:14:47.627676 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 12744
I20260923 18:14:47.628145 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.627807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:38973 (local address 127.0.0.1:59838)
0923 18:14:47.627949 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:47.628048 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 18:14:47.628353 12534 catalog_manager.cc:4584] TS 8641942a4e3741ba9a5b47dfc93105a4 (127.27.75.193:38973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fac9fe26bb1b4e709c53b36b89bb8366: Network error: Client connection negotiation failed: client connection to 127.27.75.193:38973: connect: Connection refused (error 111)
I20260923 18:14:47.628963 13052 ts_tablet_manager.cc:1905] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:47.631302 13052 ts_tablet_manager.cc:1918] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:47.631405 13052 log.cc:1196] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/fac9fe26bb1b4e709c53b36b89bb8366
I20260923 18:14:47.631659 13052 ts_tablet_manager.cc:1939] T fac9fe26bb1b4e709c53b36b89bb8366 P 824f1130c51c4d6fb0f05b5dc315bc30: Deleting consensus metadata
I20260923 18:14:47.633095 12533 catalog_manager.cc:4841] TS 824f1130c51c4d6fb0f05b5dc315bc30 (127.27.75.195:45565): tablet fac9fe26bb1b4e709c53b36b89bb8366 (table test-table [id=331f1bc9882e4e02be8ed2d9d9ede1c7]) successfully deleted
I20260923 18:14:47.633165 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 12879
I20260923 18:14:47.633823 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.633468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:46825 (local address 127.0.0.1:51200)
0923 18:14:47.633683 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:47.633733 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.194:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260923 18:14:47.637768 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 12514
2026-09-23T18:14:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2279 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T18:14:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:14:47Z Disabled control of system clock
I20260923 18:14:47.666600 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39111
--webserver_interface=127.27.75.254
--webserver_port=0
--builtin_ntp_servers=127.27.75.212:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:39111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:14:47.785068 13057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:47.785439 13057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:47.785480 13057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:47.789742 13057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:14:47.789824 13057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:47.789839 13057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:47.789855 13057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:14:47.789870 13057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:14:47.795032 13057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:39111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:39111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:47.796202 13057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:47.797657 13057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:47.806227 13063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:47.806231 13062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:47.806231 13065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:47.806581 13057 server_base.cc:1034] running on GCE node
I20260923 18:14:47.807206 13057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:47.808362 13057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:47.809523 13057 hybrid_clock.cc:648] HybridClock initialized: now 1790187287809497 us; error 47 us; skew 500 ppm
I20260923 18:14:47.811435 13057 webserver.cc:460] Webserver started at http://127.27.75.254:42599/ using document root <none> and password file <none>
I20260923 18:14:47.812005 13057 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:47.812075 13057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:47.812273 13057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:47.814404 13057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "cb2f8bce96aa4facb8448b583fc5f20d"
format_stamp: "Formatted at 2026-09-23 18:14:47 on dist-test-slave-wsx5"
server_key: "77050b31767dc17331134324dda4c8af"
server_key_iv: "f7f5662bd8c6e385d57efd6728fb0be8"
server_key_version: "encryptionkey@0"
I20260923 18:14:47.814981 13057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "cb2f8bce96aa4facb8448b583fc5f20d"
format_stamp: "Formatted at 2026-09-23 18:14:47 on dist-test-slave-wsx5"
server_key: "77050b31767dc17331134324dda4c8af"
server_key_iv: "f7f5662bd8c6e385d57efd6728fb0be8"
server_key_version: "encryptionkey@0"
I20260923 18:14:47.818898 13057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:14:47.821593 13071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:47.822906 13057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:47.823071 13057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "cb2f8bce96aa4facb8448b583fc5f20d"
format_stamp: "Formatted at 2026-09-23 18:14:47 on dist-test-slave-wsx5"
server_key: "77050b31767dc17331134324dda4c8af"
server_key_iv: "f7f5662bd8c6e385d57efd6728fb0be8"
server_key_version: "encryptionkey@0"
I20260923 18:14:47.823180 13057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:47.861953 13057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:47.862747 13057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:47.862939 13057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:47.869812 13057 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:39111
I20260923 18:14:47.869825 13124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:39111 every 8 connection(s)
I20260923 18:14:47.870802 13057 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:14:47.873170 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 13057
I20260923 18:14:47.873298 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:14:47.873575 27951 external_mini_cluster.cc:1428] Setting key 5d2f211b5c57eb591b39690ef78ee285
I20260923 18:14:47.874370 13125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:47.881834 13125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d: Bootstrap starting.
I20260923 18:14:47.884697 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:47.875086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.27.75.254:39111)
0923 18:14:47.875823 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:47.875843 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:47.875890 (+ 47us) server_negotiation.cc:407] Connection header received
0923 18:14:47.876689 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:47.876714 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:47.877303 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:47.877714 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:47.878190 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.879642 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.880670 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:47.881416 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:47.881575 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:47.882431 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:47.882437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:47.882462 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:47.882971 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:47.883050 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:47.883080 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:47.883181 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:47.883337 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 18:14:47.883622 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":138,"threads_started":1}
I20260923 18:14:47.884814 13125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:47.885823 13125 log.cc:824] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:47.887913 13125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d: No bootstrap required, opened a new log
I20260923 18:14:47.891471 13125 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2f8bce96aa4facb8448b583fc5f20d" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39111 } }
I20260923 18:14:47.891705 13125 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:47.891739 13125 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb2f8bce96aa4facb8448b583fc5f20d, State: Initialized, Role: FOLLOWER
I20260923 18:14:47.892156 13125 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2f8bce96aa4facb8448b583fc5f20d" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39111 } }
I20260923 18:14:47.892287 13125 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:14:47.892328 13125 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:14:47.892412 13125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:47.893515 13125 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2f8bce96aa4facb8448b583fc5f20d" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39111 } }
I20260923 18:14:47.893831 13125 leader_election.cc:302] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb2f8bce96aa4facb8448b583fc5f20d; no voters:
I20260923 18:14:47.894109 13125 leader_election.cc:288] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:14:47.894316 13130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:47.894529 13130 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [term 1 LEADER]: Becoming Leader. State: Replica: cb2f8bce96aa4facb8448b583fc5f20d, State: Running, Role: LEADER
I20260923 18:14:47.894881 13130 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2f8bce96aa4facb8448b583fc5f20d" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39111 } }
I20260923 18:14:47.895368 13125 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:14:47.896543 13131 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb2f8bce96aa4facb8448b583fc5f20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2f8bce96aa4facb8448b583fc5f20d" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39111 } } }
I20260923 18:14:47.896561 13132 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb2f8bce96aa4facb8448b583fc5f20d. Latest consensus state: current_term: 1 leader_uuid: "cb2f8bce96aa4facb8448b583fc5f20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2f8bce96aa4facb8448b583fc5f20d" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 39111 } } }
I20260923 18:14:47.896701 13131 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:47.896705 13132 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:47.897208 13140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:14:47.900049 13140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:14:47.905957 13140 catalog_manager.cc:1347] Generated new cluster ID: 335136b17c464e5ba88b08b19f13934a
I20260923 18:14:47.906047 13140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:14:47.923669 13140 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:14:47.924685 13140 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:14:47.930395 13140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d: Generated new TSK 0
I20260923 18:14:47.931123 13140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:14:47.940516 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39111
--builtin_ntp_servers=127.27.75.212:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:14:48.085898 13149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:48.086192 13149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:48.086230 13149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:48.090308 13149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:48.090375 13149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:48.090431 13149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:14:48.096202 13149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:48.097374 13149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:48.098731 13149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:48.108449 13157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:48.108449 13154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:48.108574 13155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:48.108942 13149 server_base.cc:1034] running on GCE node
I20260923 18:14:48.109400 13149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:48.110105 13149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:48.111280 13149 hybrid_clock.cc:648] HybridClock initialized: now 1790187288111254 us; error 36 us; skew 500 ppm
I20260923 18:14:48.113286 13149 webserver.cc:460] Webserver started at http://127.27.75.193:46345/ using document root <none> and password file <none>
I20260923 18:14:48.113852 13149 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:48.113911 13149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:48.114094 13149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:48.116345 13149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "f21c3d7b123647b7b0fb983c777b261e"
format_stamp: "Formatted at 2026-09-23 18:14:48 on dist-test-slave-wsx5"
server_key: "e344a46ec3d77c70cea3f0746305e5f4"
server_key_iv: "6ca403a40ea892029b09c6961926706f"
server_key_version: "encryptionkey@0"
I20260923 18:14:48.116906 13149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "f21c3d7b123647b7b0fb983c777b261e"
format_stamp: "Formatted at 2026-09-23 18:14:48 on dist-test-slave-wsx5"
server_key: "e344a46ec3d77c70cea3f0746305e5f4"
server_key_iv: "6ca403a40ea892029b09c6961926706f"
server_key_version: "encryptionkey@0"
I20260923 18:14:48.120594 13149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:14:48.123109 13163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:48.124410 13149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:14:48.124596 13149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "f21c3d7b123647b7b0fb983c777b261e"
format_stamp: "Formatted at 2026-09-23 18:14:48 on dist-test-slave-wsx5"
server_key: "e344a46ec3d77c70cea3f0746305e5f4"
server_key_iv: "6ca403a40ea892029b09c6961926706f"
server_key_version: "encryptionkey@0"
I20260923 18:14:48.124735 13149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:48.135362 13149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:48.136061 13149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:48.136229 13149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:48.136834 13149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:48.137832 13149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:48.137881 13149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:48.137923 13149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:48.137948 13149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:48.146082 13149 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36647
I20260923 18:14:48.146133 13277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36647 every 8 connection(s)
I20260923 18:14:48.147015 13149 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:14:48.148567 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 13149
I20260923 18:14:48.148679 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:14:48.148988 27951 external_mini_cluster.cc:1428] Setting key c96e8e44e9fd565ae489da5e492fcfde
I20260923 18:14:48.150998 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39111
--builtin_ntp_servers=127.27.75.212:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:14:48.157627 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.148852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:58399 (local address 127.27.75.254:39111)
0923 18:14:48.149014 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:48.149018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:48.150256 (+ 1238us) server_negotiation.cc:407] Connection header received
0923 18:14:48.151822 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:48.151825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:48.151895 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:48.151972 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:48.152954 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.153800 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.155507 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.155769 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.155897 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.157092 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:48.157096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:48.157099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:48.157115 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:48.157162 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:48.157172 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:48.157239 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:48.157394 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 18:14:48.157471 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:14:48.158524 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.149345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39111 (local address 127.27.75.193:58399)
0923 18:14:48.150105 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:48.150162 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:48.151602 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:48.152167 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:48.152175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:48.152267 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:48.152800 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.152814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.153989 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.153993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.155350 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.155363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.155969 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.155974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.156148 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.156799 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:14:48.156801 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:48.156845 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:14:48.156918 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:48.157252 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:48.157263 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:48.157269 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:14:48.157384 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 18:14:48.157690 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":144,"threads_started":1}
I20260923 18:14:48.159947 13278 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39111
I20260923 18:14:48.160231 13278 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:48.161067 13278 heartbeater.cc:511] Master 127.27.75.254:39111 requested a full tablet report, sending...
I20260923 18:14:48.162820 13088 ts_manager.cc:194] Registered new tserver with Master: f21c3d7b123647b7b0fb983c777b261e (127.27.75.193:36647)
I20260923 18:14:48.164017 13088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.193:58399
W20260923 18:14:48.266759 13282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:48.267071 13282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:48.267113 13282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:48.271365 13282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:48.271443 13282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:48.271503 13282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:14:48.276518 13282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:48.277717 13282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:48.279214 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:48.289031 13290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:48.289026 13288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:48.289315 13282 server_base.cc:1034] running on GCE node
W20260923 18:14:48.289052 13287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:48.289865 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:48.290599 13282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:48.291774 13282 hybrid_clock.cc:648] HybridClock initialized: now 1790187288291715 us; error 73 us; skew 500 ppm
I20260923 18:14:48.293829 13282 webserver.cc:460] Webserver started at http://127.27.75.194:37151/ using document root <none> and password file <none>
I20260923 18:14:48.294406 13282 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:48.294479 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:48.294684 13282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:48.296900 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "98dc44ea92974187b1bfe9eadb558bb8"
format_stamp: "Formatted at 2026-09-23 18:14:48 on dist-test-slave-wsx5"
server_key: "6f3a487ca5f323718e2152ac129a3b19"
server_key_iv: "a7104726b7997fc5282d350d1a4b550c"
server_key_version: "encryptionkey@0"
I20260923 18:14:48.297555 13282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "98dc44ea92974187b1bfe9eadb558bb8"
format_stamp: "Formatted at 2026-09-23 18:14:48 on dist-test-slave-wsx5"
server_key: "6f3a487ca5f323718e2152ac129a3b19"
server_key_iv: "a7104726b7997fc5282d350d1a4b550c"
server_key_version: "encryptionkey@0"
I20260923 18:14:48.301519 13282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:14:48.304157 13296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:48.305688 13282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:48.305828 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "98dc44ea92974187b1bfe9eadb558bb8"
format_stamp: "Formatted at 2026-09-23 18:14:48 on dist-test-slave-wsx5"
server_key: "6f3a487ca5f323718e2152ac129a3b19"
server_key_iv: "a7104726b7997fc5282d350d1a4b550c"
server_key_version: "encryptionkey@0"
I20260923 18:14:48.305940 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:48.322384 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:48.323110 13282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:48.323287 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:48.323879 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:48.324838 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:48.324887 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:48.324940 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:48.324965 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:48.333899 13282 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:43151
I20260923 18:14:48.333985 13409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:43151 every 8 connection(s)
I20260923 18:14:48.334851 13282 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:14:48.339133 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 13282
I20260923 18:14:48.339236 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:14:48.339509 27951 external_mini_cluster.cc:1428] Setting key 451062568fd9095ba40b788638b01133
I20260923 18:14:48.341694 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39111
--builtin_ntp_servers=127.27.75.212:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:14:48.345017 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.336840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:45631 (local address 127.27.75.254:39111)
0923 18:14:48.336994 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:48.336998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:48.338200 (+ 1202us) server_negotiation.cc:407] Connection header received
0923 18:14:48.339646 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:48.339650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:48.339712 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:48.339817 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:48.341066 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.341639 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.342897 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.343117 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.343209 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.344436 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:48.344439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:48.344441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:48.344456 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:48.344480 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:48.344488 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:48.344550 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:48.344810 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 18:14:48.344887 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:14:48.346210 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.337225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39111 (local address 127.27.75.194:45631)
0923 18:14:48.337993 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:48.338040 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:48.339403 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:48.340086 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:48.340097 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:48.340218 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:48.340904 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.340922 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.341773 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.341776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.342761 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.342772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.343280 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.343285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.343463 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.344082 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:14:48.344084 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:48.344142 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:14:48.344232 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:48.344566 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:48.344573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:48.344579 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:14:48.344707 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 18:14:48.345045 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":123,"threads_started":1}
I20260923 18:14:48.347590 13410 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39111
I20260923 18:14:48.347842 13410 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:48.348421 13410 heartbeater.cc:511] Master 127.27.75.254:39111 requested a full tablet report, sending...
I20260923 18:14:48.349520 13088 ts_manager.cc:194] Registered new tserver with Master: 98dc44ea92974187b1bfe9eadb558bb8 (127.27.75.194:43151)
I20260923 18:14:48.350075 13088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.194:45631
W20260923 18:14:48.454634 13413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:48.454932 13413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:48.454988 13413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:48.459141 13413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:48.459203 13413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:48.459281 13413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:14:48.464238 13413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:39111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:48.465404 13413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:48.466789 13413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:48.476773 13419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:48.476773 13418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:48.476878 13421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:48.477175 13413 server_base.cc:1034] running on GCE node
I20260923 18:14:48.477614 13413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:48.478325 13413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:48.479525 13413 hybrid_clock.cc:648] HybridClock initialized: now 1790187288479432 us; error 105 us; skew 500 ppm
I20260923 18:14:48.481655 13413 webserver.cc:460] Webserver started at http://127.27.75.195:38651/ using document root <none> and password file <none>
I20260923 18:14:48.482244 13413 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:48.482317 13413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:48.482510 13413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:48.484701 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "66f10b2cae9d427bbafd2339c4058001"
format_stamp: "Formatted at 2026-09-23 18:14:48 on dist-test-slave-wsx5"
server_key: "8182cb2fbe00c0082dca68eb552f1e0a"
server_key_iv: "01e7e2e592eb650822117512fda71ada"
server_key_version: "encryptionkey@0"
I20260923 18:14:48.485309 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "66f10b2cae9d427bbafd2339c4058001"
format_stamp: "Formatted at 2026-09-23 18:14:48 on dist-test-slave-wsx5"
server_key: "8182cb2fbe00c0082dca68eb552f1e0a"
server_key_iv: "01e7e2e592eb650822117512fda71ada"
server_key_version: "encryptionkey@0"
I20260923 18:14:48.489013 13413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:14:48.491534 13428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:48.492861 13413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 18:14:48.492985 13413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "66f10b2cae9d427bbafd2339c4058001"
format_stamp: "Formatted at 2026-09-23 18:14:48 on dist-test-slave-wsx5"
server_key: "8182cb2fbe00c0082dca68eb552f1e0a"
server_key_iv: "01e7e2e592eb650822117512fda71ada"
server_key_version: "encryptionkey@0"
I20260923 18:14:48.493088 13413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:48.503861 13413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:48.504592 13413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:48.504793 13413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:48.505470 13413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:48.506495 13413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:48.506542 13413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:48.506599 13413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:48.506624 13413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:48.515013 13413 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:34373
I20260923 18:14:48.515062 13542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:34373 every 8 connection(s)
I20260923 18:14:48.515985 13413 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:14:48.518121 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 13413
I20260923 18:14:48.518213 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:14:48.518465 27951 external_mini_cluster.cc:1428] Setting key aba8e105942aea2207e042c17f053420
I20260923 18:14:48.525154 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.517924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:43529 (local address 127.27.75.254:39111)
0923 18:14:48.518082 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:48.518087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:48.519184 (+ 1097us) server_negotiation.cc:407] Connection header received
0923 18:14:48.520673 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:48.520675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:48.520720 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:48.520782 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:48.521676 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.522231 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.523345 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.523590 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.523670 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.524753 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:48.524756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:48.524758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:48.524768 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:48.524786 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:48.524793 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:48.524843 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:48.524956 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 18:14:48.525010 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:14:48.525987 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.518348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:39111 (local address 127.27.75.195:43529)
0923 18:14:48.519036 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:48.519088 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:48.520498 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:48.520922 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:48.520929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:48.521014 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:48.521550 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.521564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.522353 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.522356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.523232 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.523242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.523758 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.523761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.523916 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.524517 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:14:48.524519 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:48.524564 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:14:48.524629 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:48.524845 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:48.524850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:48.524854 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 18:14:48.524948 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 18:14:48.525198 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":109,"threads_started":1}
I20260923 18:14:48.527096 13543 heartbeater.cc:347] Connected to a master server at 127.27.75.254:39111
I20260923 18:14:48.527352 13543 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:48.527959 13543 heartbeater.cc:511] Master 127.27.75.254:39111 requested a full tablet report, sending...
I20260923 18:14:48.529049 13088 ts_manager.cc:194] Registered new tserver with Master: 66f10b2cae9d427bbafd2339c4058001 (127.27.75.195:34373)
I20260923 18:14:48.529675 13088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.195:43529
I20260923 18:14:48.532687 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:14:48.538535 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.534784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.27.75.254:39111)
0923 18:14:48.534952 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:48.534956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:48.535070 (+ 114us) server_negotiation.cc:407] Connection header received
0923 18:14:48.535189 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:48.535192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:48.535235 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:48.535329 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:48.535603 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.536132 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.536988 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.537263 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.537352 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.538110 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:48.538114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:48.538117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:48.538130 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:48.538152 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:48.538161 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:48.538217 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:48.538325 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 18:14:48.538390 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 18:14:48.541831 13088 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:14:48.544866 13088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:14:48.560316 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.554741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.27.75.193:36647)
0923 18:14:48.555628 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:48.555639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:48.555661 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:14:48.555732 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:48.555737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:48.555910 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:48.556047 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:48.556249 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.557341 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.558670 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.559478 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.559592 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.559794 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:48.560024 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 18:14:48.560137 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":93,"threads_started":1}
I20260923 18:14:48.560345 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.554209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36647 (local address 127.0.0.1:37900)
0923 18:14:48.555100 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:48.555113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:48.555198 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:48.556037 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:48.556040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:48.556055 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:48.556141 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.556147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.557443 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.557446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.558538 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.558547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.559630 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.559635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.559867 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.559886 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 18:14:48.559982 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 18:14:48.560172 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":125,"thread_start_us":121,"threads_started":1}
I20260923 18:14:48.560345 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.553956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:34373 (local address 127.0.0.1:45052)
0923 18:14:48.554291 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:48.554340 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:48.554468 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:48.555456 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:48.555463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:48.555487 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:48.555642 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.555650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.557286 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.557289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.558526 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.558532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.559630 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.559635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.559805 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.559855 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 18:14:48.559982 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 18:14:48.560172 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":113,"thread_start_us":89,"threads_started":1}
I20260923 18:14:48.560894 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.554151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.27.75.195:34373)
0923 18:14:48.554946 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:48.554958 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:48.554983 (+ 25us) server_negotiation.cc:407] Connection header received
0923 18:14:48.555061 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:48.555068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:48.555282 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:48.555800 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:48.555954 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.557174 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.558635 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.559478 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.559592 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.559794 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:48.560597 (+ 803us) server_negotiation.cc:299] Negotiation successful
0923 18:14:48.560719 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":171,"threads_started":1}
I20260923 18:14:48.561519 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.554693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:43151 (local address 127.0.0.1:58004)
0923 18:14:48.555380 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:48.555392 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:48.555468 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:48.555846 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:48.555849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:48.555865 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:48.555971 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.555977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.556984 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.556987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.558499 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.558511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.561081 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.561086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.561205 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.561231 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 18:14:48.561296 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 18:14:48.561386 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":31,"thread_start_us":92,"threads_started":1}
I20260923 18:14:48.562153 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.554995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.27.75.194:43151)
0923 18:14:48.555397 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:48.555415 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:48.555438 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:14:48.555581 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:48.555585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:48.555727 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:48.555853 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:48.556087 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.556878 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.559864 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.560640 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.560741 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.561843 (+ 1102us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:48.561898 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 18:14:48.561993 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":117,"threads_started":1}
I20260923 18:14:48.563750 13476 tablet_service.cc:1467] Processing CreateTablet for tablet c3fd7c74833f454f8679f0085d848237 (DEFAULT_TABLE table=test-table [id=33838b0047fd4e8188b8af82e498af5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:48.563761 13211 tablet_service.cc:1467] Processing CreateTablet for tablet c3fd7c74833f454f8679f0085d848237 (DEFAULT_TABLE table=test-table [id=33838b0047fd4e8188b8af82e498af5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:48.564956 13476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3fd7c74833f454f8679f0085d848237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:48.564956 13211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3fd7c74833f454f8679f0085d848237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:48.567130 13344 tablet_service.cc:1467] Processing CreateTablet for tablet c3fd7c74833f454f8679f0085d848237 (DEFAULT_TABLE table=test-table [id=33838b0047fd4e8188b8af82e498af5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:48.568259 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3fd7c74833f454f8679f0085d848237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:48.573665 13560 tablet_bootstrap.cc:492] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001: Bootstrap starting.
I20260923 18:14:48.574137 13561 tablet_bootstrap.cc:492] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e: Bootstrap starting.
I20260923 18:14:48.574934 13562 tablet_bootstrap.cc:492] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: Bootstrap starting.
I20260923 18:14:48.576570 13561 tablet_bootstrap.cc:654] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:48.576740 13562 tablet_bootstrap.cc:654] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:48.577361 13560 tablet_bootstrap.cc:654] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:48.577453 13562 log.cc:824] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:48.577468 13561 log.cc:824] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:48.578150 13560 log.cc:824] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:48.579444 13562 tablet_bootstrap.cc:492] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: No bootstrap required, opened a new log
I20260923 18:14:48.579509 13561 tablet_bootstrap.cc:492] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e: No bootstrap required, opened a new log
I20260923 18:14:48.579650 13562 ts_tablet_manager.cc:1397] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 18:14:48.579703 13561 ts_tablet_manager.cc:1397] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:14:48.580400 13560 tablet_bootstrap.cc:492] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001: No bootstrap required, opened a new log
I20260923 18:14:48.580633 13560 ts_tablet_manager.cc:1397] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:14:48.582988 13561 raft_consensus.cc:348] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f10b2cae9d427bbafd2339c4058001" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34373 } } peers { permanent_uuid: "98dc44ea92974187b1bfe9eadb558bb8" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43151 } } peers { permanent_uuid: "f21c3d7b123647b7b0fb983c777b261e" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36647 } }
I20260923 18:14:48.583237 13561 raft_consensus.cc:374] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:48.583287 13561 raft_consensus.cc:729] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c3d7b123647b7b0fb983c777b261e, State: Initialized, Role: FOLLOWER
I20260923 18:14:48.583750 13561 consensus_queue.cc:260] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f10b2cae9d427bbafd2339c4058001" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34373 } } peers { permanent_uuid: "98dc44ea92974187b1bfe9eadb558bb8" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43151 } } peers { permanent_uuid: "f21c3d7b123647b7b0fb983c777b261e" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36647 } }
I20260923 18:14:48.583971 13562 raft_consensus.cc:348] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f10b2cae9d427bbafd2339c4058001" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34373 } } peers { permanent_uuid: "98dc44ea92974187b1bfe9eadb558bb8" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43151 } } peers { permanent_uuid: "f21c3d7b123647b7b0fb983c777b261e" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36647 } }
I20260923 18:14:48.584241 13562 raft_consensus.cc:374] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:48.584299 13562 raft_consensus.cc:729] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98dc44ea92974187b1bfe9eadb558bb8, State: Initialized, Role: FOLLOWER
I20260923 18:14:48.584573 13278 heartbeater.cc:503] Master 127.27.75.254:39111 was elected leader, sending a full tablet report...
I20260923 18:14:48.584743 13562 consensus_queue.cc:260] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f10b2cae9d427bbafd2339c4058001" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34373 } } peers { permanent_uuid: "98dc44ea92974187b1bfe9eadb558bb8" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43151 } } peers { permanent_uuid: "f21c3d7b123647b7b0fb983c777b261e" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36647 } }
I20260923 18:14:48.584791 13560 raft_consensus.cc:348] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f10b2cae9d427bbafd2339c4058001" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34373 } } peers { permanent_uuid: "98dc44ea92974187b1bfe9eadb558bb8" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43151 } } peers { permanent_uuid: "f21c3d7b123647b7b0fb983c777b261e" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36647 } }
I20260923 18:14:48.585034 13560 raft_consensus.cc:374] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:48.585091 13560 raft_consensus.cc:729] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f10b2cae9d427bbafd2339c4058001, State: Initialized, Role: FOLLOWER
I20260923 18:14:48.585588 13561 ts_tablet_manager.cc:1428] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:14:48.585579 13560 consensus_queue.cc:260] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f10b2cae9d427bbafd2339c4058001" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34373 } } peers { permanent_uuid: "98dc44ea92974187b1bfe9eadb558bb8" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43151 } } peers { permanent_uuid: "f21c3d7b123647b7b0fb983c777b261e" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36647 } }
I20260923 18:14:48.585790 13562 ts_tablet_manager.cc:1428] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260923 18:14:48.585871 13411 tablet.cc:2369] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:14:48.586148 13410 heartbeater.cc:503] Master 127.27.75.254:39111 was elected leader, sending a full tablet report...
I20260923 18:14:48.586525 13560 ts_tablet_manager.cc:1428] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260923 18:14:48.586797 13543 heartbeater.cc:503] Master 127.27.75.254:39111 was elected leader, sending a full tablet report...
I20260923 18:14:48.631165 13567 raft_consensus.cc:482] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:48.631367 13567 raft_consensus.cc:504] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f10b2cae9d427bbafd2339c4058001" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34373 } } peers { permanent_uuid: "98dc44ea92974187b1bfe9eadb558bb8" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43151 } } peers { permanent_uuid: "f21c3d7b123647b7b0fb983c777b261e" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36647 } }
I20260923 18:14:48.632609 13567 leader_election.cc:288] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66f10b2cae9d427bbafd2339c4058001 (127.27.75.195:34373), f21c3d7b123647b7b0fb983c777b261e (127.27.75.193:36647)
I20260923 18:14:48.636781 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.632833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:34373 (local address 127.27.75.194:50257)
0923 18:14:48.632958 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:48.632974 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:48.633087 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:48.633487 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:48.633490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:48.633504 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:48.633632 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.633639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.634400 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.634404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.635614 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.635625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.636392 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.636395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.636471 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.636491 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 18:14:48.636563 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:14:48.636637 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 18:14:48.636808 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.633085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:50257 (local address 127.27.75.195:34373)
0923 18:14:48.633276 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:48.633279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:48.633291 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:14:48.633340 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:48.633343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:48.633380 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:48.633449 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:48.633751 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.634257 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.635742 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.636264 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.636387 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.636438 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:48.636599 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 18:14:48.636677 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260923 18:14:48.637048 13569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.632903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36647 (local address 127.27.75.194:36867)
0923 18:14:48.633341 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:48.633352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:48.633457 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:48.633768 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:48.633771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:48.633787 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:48.633900 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.633905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.634824 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.634827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.635870 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:48.635877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.636665 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.636668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:48.636758 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.636778 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 18:14:48.636851 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 18:14:48.636917 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":136,"threads_started":1}
I20260923 18:14:48.637072 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.633021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:36867 (local address 127.27.75.193:36647)
0923 18:14:48.633166 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:48.633170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:48.633441 (+ 271us) server_negotiation.cc:407] Connection header received
0923 18:14:48.633626 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:48.633628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:48.633675 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:48.633743 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:48.634017 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.634706 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.635989 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.636536 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.636654 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.636697 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:48.636885 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 18:14:48.636960 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:14:48.637698 13231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3fd7c74833f454f8679f0085d848237" candidate_uuid: "98dc44ea92974187b1bfe9eadb558bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c3d7b123647b7b0fb983c777b261e" is_pre_election: true
I20260923 18:14:48.637717 13496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3fd7c74833f454f8679f0085d848237" candidate_uuid: "98dc44ea92974187b1bfe9eadb558bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f10b2cae9d427bbafd2339c4058001" is_pre_election: true
I20260923 18:14:48.637979 13496 raft_consensus.cc:2454] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98dc44ea92974187b1bfe9eadb558bb8 in term 0.
I20260923 18:14:48.637987 13231 raft_consensus.cc:2454] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98dc44ea92974187b1bfe9eadb558bb8 in term 0.
I20260923 18:14:48.638444 13299 leader_election.cc:302] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98dc44ea92974187b1bfe9eadb558bb8, f21c3d7b123647b7b0fb983c777b261e; no voters:
I20260923 18:14:48.638723 13567 raft_consensus.cc:2781] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:14:48.638809 13567 raft_consensus.cc:482] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:14:48.638856 13567 raft_consensus.cc:3037] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:48.639856 13567 raft_consensus.cc:504] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f10b2cae9d427bbafd2339c4058001" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34373 } } peers { permanent_uuid: "98dc44ea92974187b1bfe9eadb558bb8" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43151 } } peers { permanent_uuid: "f21c3d7b123647b7b0fb983c777b261e" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36647 } }
I20260923 18:14:48.640194 13567 leader_election.cc:288] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [CANDIDATE]: Term 1 election: Requested vote from peers 66f10b2cae9d427bbafd2339c4058001 (127.27.75.195:34373), f21c3d7b123647b7b0fb983c777b261e (127.27.75.193:36647)
I20260923 18:14:48.640549 13496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3fd7c74833f454f8679f0085d848237" candidate_uuid: "98dc44ea92974187b1bfe9eadb558bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f10b2cae9d427bbafd2339c4058001"
I20260923 18:14:48.640558 13231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3fd7c74833f454f8679f0085d848237" candidate_uuid: "98dc44ea92974187b1bfe9eadb558bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c3d7b123647b7b0fb983c777b261e"
I20260923 18:14:48.640666 13496 raft_consensus.cc:3037] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:48.640681 13231 raft_consensus.cc:3037] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:48.641845 13496 raft_consensus.cc:2454] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98dc44ea92974187b1bfe9eadb558bb8 in term 1.
I20260923 18:14:48.641896 13231 raft_consensus.cc:2454] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98dc44ea92974187b1bfe9eadb558bb8 in term 1.
I20260923 18:14:48.642196 13297 leader_election.cc:302] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f10b2cae9d427bbafd2339c4058001, 98dc44ea92974187b1bfe9eadb558bb8; no voters:
I20260923 18:14:48.642382 13567 raft_consensus.cc:2781] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:48.642578 13567 raft_consensus.cc:686] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 1 LEADER]: Becoming Leader. State: Replica: 98dc44ea92974187b1bfe9eadb558bb8, State: Running, Role: LEADER
I20260923 18:14:48.642928 13567 consensus_queue.cc:237] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f10b2cae9d427bbafd2339c4058001" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34373 } } peers { permanent_uuid: "98dc44ea92974187b1bfe9eadb558bb8" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43151 } } peers { permanent_uuid: "f21c3d7b123647b7b0fb983c777b261e" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36647 } }
I20260923 18:14:48.646407 13088 catalog_manager.cc:5495] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98dc44ea92974187b1bfe9eadb558bb8 (127.27.75.194). New cstate: current_term: 1 leader_uuid: "98dc44ea92974187b1bfe9eadb558bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f10b2cae9d427bbafd2339c4058001" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98dc44ea92974187b1bfe9eadb558bb8" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f21c3d7b123647b7b0fb983c777b261e" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36647 } health_report { overall_health: UNKNOWN } } }
W20260923 18:14:48.648183 13280 tablet.cc:2369] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:14:48.662806 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.659036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.27.75.194:43151)
0923 18:14:48.659184 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:48.659188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:48.659203 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:14:48.659338 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:48.659341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:48.659405 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:48.659492 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:14:48.659681 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.660166 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.661030 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:48.661302 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:48.661431 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:48.661787 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:14:48.662427 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:14:48.662521 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:48.662581 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 18:14:48.662633 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 18:14:48.668669 13496 raft_consensus.cc:1261] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001 [term 1 FOLLOWER]: Refusing update from remote peer 98dc44ea92974187b1bfe9eadb558bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:48.668669 13231 raft_consensus.cc:1261] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e [term 1 FOLLOWER]: Refusing update from remote peer 98dc44ea92974187b1bfe9eadb558bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:48.669519 13570 consensus_queue.cc:1035] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f10b2cae9d427bbafd2339c4058001" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:48.669710 13567 consensus_queue.cc:1035] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c3d7b123647b7b0fb983c777b261e" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:48.678191 13575 mvcc.cc:205] Tried to move back new op lower bound from 7332607134378962944 to 7332607134284083200. Current Snapshot: MvccSnapshot[applied={T|T < 7332607134378962944}]
I20260923 18:14:48.679080 13576 mvcc.cc:205] Tried to move back new op lower bound from 7332607134378962944 to 7332607134284083200. Current Snapshot: MvccSnapshot[applied={T|T < 7332607134378962944}]
I20260923 18:14:48.687474 13086 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:48.687631 13086 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:48.690325 13086 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb2f8bce96aa4facb8448b583fc5f20d: Sending DeleteTablet for 3 replicas of tablet c3fd7c74833f454f8679f0085d848237
I20260923 18:14:48.691267 13344 tablet_service.cc:1514] Processing DeleteTablet for tablet c3fd7c74833f454f8679f0085d848237 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:48 UTC) from {username='slave'} at 127.0.0.1:58004
I20260923 18:14:48.691263 13476 tablet_service.cc:1514] Processing DeleteTablet for tablet c3fd7c74833f454f8679f0085d848237 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:48 UTC) from {username='slave'} at 127.0.0.1:45052
I20260923 18:14:48.691847 13586 tablet_replica.cc:330] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001: stopping tablet replica
I20260923 18:14:48.691835 13211 tablet_service.cc:1514] Processing DeleteTablet for tablet c3fd7c74833f454f8679f0085d848237 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:48 UTC) from {username='slave'} at 127.0.0.1:37900
I20260923 18:14:48.692233 13586 raft_consensus.cc:2229] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:48.692262 13587 tablet_replica.cc:330] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e: stopping tablet replica
I20260923 18:14:48.692605 13586 raft_consensus.cc:2258] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:48.693044 13585 tablet_replica.cc:330] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: stopping tablet replica
I20260923 18:14:48.693063 13587 raft_consensus.cc:2229] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:48.693369 13585 raft_consensus.cc:2229] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:14:48.693595 13587 raft_consensus.cc:2258] T c3fd7c74833f454f8679f0085d848237 P f21c3d7b123647b7b0fb983c777b261e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:48.693785 13585 raft_consensus.cc:2258] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:48.694442 13586 ts_tablet_manager.cc:1905] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:48.694639 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 13149
I20260923 18:14:48.695950 13585 ts_tablet_manager.cc:1905] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:48.697393 13586 ts_tablet_manager.cc:1918] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:48.697499 13586 log.cc:1196] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/c3fd7c74833f454f8679f0085d848237
I20260923 18:14:48.697794 13586 ts_tablet_manager.cc:1939] T c3fd7c74833f454f8679f0085d848237 P 66f10b2cae9d427bbafd2339c4058001: Deleting consensus metadata
I20260923 18:14:48.698558 13585 ts_tablet_manager.cc:1918] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:48.698647 13585 log.cc:1196] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/c3fd7c74833f454f8679f0085d848237
I20260923 18:14:48.698899 13585 ts_tablet_manager.cc:1939] T c3fd7c74833f454f8679f0085d848237 P 98dc44ea92974187b1bfe9eadb558bb8: Deleting consensus metadata
I20260923 18:14:48.699155 13072 catalog_manager.cc:4841] TS 66f10b2cae9d427bbafd2339c4058001 (127.27.75.195:34373): tablet c3fd7c74833f454f8679f0085d848237 (table test-table [id=33838b0047fd4e8188b8af82e498af5a]) successfully deleted
I20260923 18:14:48.700392 13072 catalog_manager.cc:4841] TS 98dc44ea92974187b1bfe9eadb558bb8 (127.27.75.194:43151): tablet c3fd7c74833f454f8679f0085d848237 (table test-table [id=33838b0047fd4e8188b8af82e498af5a]) successfully deleted
W20260923 18:14:48.704190 13074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:36647 (error 108)
I20260923 18:14:48.704347 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 13282
I20260923 18:14:48.706138 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:48.705811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36647 (local address 127.0.0.1:37902)
0923 18:14:48.705964 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:48.706040 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 18:14:48.706331 13074 catalog_manager.cc:4584] TS f21c3d7b123647b7b0fb983c777b261e (127.27.75.193:36647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3fd7c74833f454f8679f0085d848237: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36647: connect: Connection refused (error 111)
I20260923 18:14:48.709865 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 13413
I20260923 18:14:48.714879 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 13057
2026-09-23T18:14:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1076 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:14:48 dist-test-slave-wsx5 krb5kdc[13591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:14:48 dist-test-slave-wsx5 krb5kdc[13591](info): set up 2 sockets
Sep 23 18:14:48 dist-test-slave-wsx5 krb5kdc[13591](info): commencing operation
krb5kdc: starting...
W20260923 18:14:49.683079 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:14:49 dist-test-slave-wsx5 krb5kdc[13591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:14:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:14:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:14:49.808102 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42789
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:42789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:49.921329 13604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:49.921643 13604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:49.921689 13604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:49.925876 13604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:14:49.925948 13604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:49.925968 13604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:49.925985 13604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:14:49.925999 13604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:14:49.931583 13604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:42789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:49.932885 13604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:49.934347 13604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:49.944038 13612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:49.944038 13610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:49.944082 13609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:49.944521 13604 server_base.cc:1034] running on GCE node
I20260923 18:14:49.945195 13604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:49.946426 13604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:49.947638 13604 hybrid_clock.cc:648] HybridClock initialized: now 1790187289947619 us; error 37 us; skew 500 ppm
Sep 23 18:14:49 dist-test-slave-wsx5 krb5kdc[13591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:49.960727 13604 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:14:49.961844 13604 webserver.cc:460] Webserver started at http://127.27.75.254:37019/ using document root <none> and password file <none>
I20260923 18:14:49.962420 13604 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:49.962492 13604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:49.962687 13604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:49.964823 13604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "c387f31b45674788bfdaf29779e480c7"
format_stamp: "Formatted at 2026-09-23 18:14:49 on dist-test-slave-wsx5"
server_key: "50c31ac2c0378e4907daf32b8a363b11"
server_key_iv: "338a684e8a6c859569453f96ba58e7df"
server_key_version: "encryptionkey@0"
I20260923 18:14:49.965386 13604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "c387f31b45674788bfdaf29779e480c7"
format_stamp: "Formatted at 2026-09-23 18:14:49 on dist-test-slave-wsx5"
server_key: "50c31ac2c0378e4907daf32b8a363b11"
server_key_iv: "338a684e8a6c859569453f96ba58e7df"
server_key_version: "encryptionkey@0"
I20260923 18:14:49.969352 13604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:14:49.971947 13619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:49.973451 13604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:49.973575 13604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "c387f31b45674788bfdaf29779e480c7"
format_stamp: "Formatted at 2026-09-23 18:14:49 on dist-test-slave-wsx5"
server_key: "50c31ac2c0378e4907daf32b8a363b11"
server_key_iv: "338a684e8a6c859569453f96ba58e7df"
server_key_version: "encryptionkey@0"
I20260923 18:14:49.973685 13604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:49.984709 13604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:49.987856 13604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:49.988058 13604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:49.994691 13604 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:42789
I20260923 18:14:49.994706 13671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:42789 every 8 connection(s)
I20260923 18:14:49.995666 13604 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:14:49.998524 13672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:50.004089 13672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7: Bootstrap starting.
I20260923 18:14:50.004940 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 13604
I20260923 18:14:50.005080 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:14:50.005429 27951 external_mini_cluster.cc:1428] Setting key 7ae930e8ea1da4632df0d901a01c113b
I20260923 18:14:50.007434 13672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:50.008217 13672 log.cc:824] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:50.010818 13672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7: No bootstrap required, opened a new log
I20260923 18:14:50.015683 13672 raft_consensus.cc:348] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c387f31b45674788bfdaf29779e480c7" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42789 } }
I20260923 18:14:50.015969 13672 raft_consensus.cc:374] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:50.016016 13672 raft_consensus.cc:729] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c387f31b45674788bfdaf29779e480c7, State: Initialized, Role: FOLLOWER
Sep 23 18:14:50 dist-test-slave-wsx5 krb5kdc[13591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:50.016512 13672 consensus_queue.cc:260] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c387f31b45674788bfdaf29779e480c7" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42789 } }
I20260923 18:14:50.016661 13672 raft_consensus.cc:388] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:14:50.016731 13672 raft_consensus.cc:482] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:14:50.016834 13672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:50.018106 13672 raft_consensus.cc:504] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c387f31b45674788bfdaf29779e480c7" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42789 } }
I20260923 18:14:50.018553 13672 leader_election.cc:302] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c387f31b45674788bfdaf29779e480c7; no voters:
I20260923 18:14:50.018936 13672 leader_election.cc:288] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:14:50.019143 13677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:50.019574 13677 raft_consensus.cc:686] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [term 1 LEADER]: Becoming Leader. State: Replica: c387f31b45674788bfdaf29779e480c7, State: Running, Role: LEADER
I20260923 18:14:50.019965 13677 consensus_queue.cc:237] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c387f31b45674788bfdaf29779e480c7" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42789 } }
I20260923 18:14:50.020406 13672 sys_catalog.cc:564] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:14:50.021682 13678 sys_catalog.cc:455] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c387f31b45674788bfdaf29779e480c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c387f31b45674788bfdaf29779e480c7" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42789 } } }
I20260923 18:14:50.021809 13678 sys_catalog.cc:458] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:50.022135 13679 sys_catalog.cc:455] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c387f31b45674788bfdaf29779e480c7. Latest consensus state: current_term: 1 leader_uuid: "c387f31b45674788bfdaf29779e480c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c387f31b45674788bfdaf29779e480c7" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42789 } } }
I20260923 18:14:50.022213 13679 sys_catalog.cc:458] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7 [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:50.022435 13686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:14:50.022929 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.006828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59076 (local address 127.27.75.254:42789)
0923 18:14:50.007363 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:50.007379 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:50.007417 (+ 38us) server_negotiation.cc:407] Connection header received
0923 18:14:50.008198 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:50.008219 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:50.008643 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:50.009072 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:50.010180 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.011261 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.012659 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.013482 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.013636 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.016963 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:50.016971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:50.016990 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:50.017030 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:50.019209 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:50.019815 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:50.019822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:50.019828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:50.019978 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:50.020334 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:50.020337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:50.020339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:50.020908 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:50.021114 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:50.021491 (+ 377us) server_negotiation.cc:299] Negotiation successful
0923 18:14:50.021837 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":158,"threads_started":1}
I20260923 18:14:50.026389 13686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:14:50.032819 13686 catalog_manager.cc:1347] Generated new cluster ID: 6544e265322e48d6915117a5d68e1372
I20260923 18:14:50.032903 13686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:14:50.049590 13686 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:14:50.050617 13686 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:14:50.056064 13686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7: Generated new TSK 0
I20260923 18:14:50.056835 13686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:14:50.088522 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42789
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:50.207669 13698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:50.207975 13698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:50.208019 13698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:50.212294 13698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:50.212387 13698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:50.212451 13698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:14:50.218904 13698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:50.220115 13698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:50.221271 13698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:50.231941 13703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:50.231856 13706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:50.231853 13704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:50.232170 13698 server_base.cc:1034] running on GCE node
I20260923 18:14:50.232702 13698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:50.233481 13698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:50.234651 13698 hybrid_clock.cc:648] HybridClock initialized: now 1790187290234618 us; error 46 us; skew 500 ppm
Sep 23 18:14:50 dist-test-slave-wsx5 krb5kdc[13591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:50.237534 13698 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:14:50.238493 13698 webserver.cc:460] Webserver started at http://127.27.75.193:45453/ using document root <none> and password file <none>
I20260923 18:14:50.239032 13698 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:50.239089 13698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:50.239259 13698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:50.241381 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "0158e9c589704e30bd90c6d1072d38e6"
format_stamp: "Formatted at 2026-09-23 18:14:50 on dist-test-slave-wsx5"
server_key: "40528b3f6583997f7fc628d87c934671"
server_key_iv: "2554fef0f3e8d62c6a9495abe9aa466c"
server_key_version: "encryptionkey@0"
I20260923 18:14:50.241928 13698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "0158e9c589704e30bd90c6d1072d38e6"
format_stamp: "Formatted at 2026-09-23 18:14:50 on dist-test-slave-wsx5"
server_key: "40528b3f6583997f7fc628d87c934671"
server_key_iv: "2554fef0f3e8d62c6a9495abe9aa466c"
server_key_version: "encryptionkey@0"
I20260923 18:14:50.245905 13698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:14:50.248747 13713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:50.250291 13698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 18:14:50.250427 13698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "0158e9c589704e30bd90c6d1072d38e6"
format_stamp: "Formatted at 2026-09-23 18:14:50 on dist-test-slave-wsx5"
server_key: "40528b3f6583997f7fc628d87c934671"
server_key_iv: "2554fef0f3e8d62c6a9495abe9aa466c"
server_key_version: "encryptionkey@0"
I20260923 18:14:50.250553 13698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:50.303133 13698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:50.306648 13698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:50.306847 13698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:50.307459 13698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:50.308327 13698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:50.308377 13698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:50.308430 13698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:50.308452 13698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:50.317077 13698 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:41719
I20260923 18:14:50.317111 13827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:41719 every 8 connection(s)
I20260923 18:14:50.318058 13698 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:14:50.326628 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 13698
I20260923 18:14:50.326785 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:14:50.327064 27951 external_mini_cluster.cc:1428] Setting key 6a78a1154fa9b35555ec02f256b96c5b
Sep 23 18:14:50 dist-test-slave-wsx5 krb5kdc[13591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:50.333277 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.320172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:54257 (local address 127.27.75.254:42789)
0923 18:14:50.320334 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:50.320337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:50.321472 (+ 1135us) server_negotiation.cc:407] Connection header received
0923 18:14:50.322893 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:50.322897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:50.322961 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:50.323040 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:50.324353 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.324900 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.326139 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.326414 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.326509 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.329398 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:50.329401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:50.329403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:50.329430 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:50.331005 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:50.331592 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:50.331595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:50.331597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:50.331681 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:50.332041 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:50.332044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:50.332045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:50.332241 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:50.332329 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:50.332939 (+ 610us) server_negotiation.cc:299] Negotiation successful
0923 18:14:50.333081 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 18:14:50.334244 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.320574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42789 (local address 127.27.75.193:54257)
0923 18:14:50.321273 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:50.321324 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:50.322662 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:50.323242 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:50.323250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:50.323640 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:50.324184 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.324199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.325035 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.325038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.326009 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.326020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.326575 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.326579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.326745 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.327374 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:50.327395 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:50.329209 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:50.331157 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:50.331162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:50.331177 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:50.331457 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:50.331770 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:50.331773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:50.331775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:50.331936 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:50.332345 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:50.332352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:50.332673 (+ 321us) client_negotiation.cc:769] Sending connection context
0923 18:14:50.332858 (+ 185us) client_negotiation.cc:240] Negotiation successful
0923 18:14:50.333294 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":174,"threads_started":1}
I20260923 18:14:50.335525 13828 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42789
I20260923 18:14:50.335754 13828 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
I20260923 18:14:50.336251 13828 heartbeater.cc:511] Master 127.27.75.254:42789 requested a full tablet report, sending...
I20260923 18:14:50.337822 13636 ts_manager.cc:194] Registered new tserver with Master: 0158e9c589704e30bd90c6d1072d38e6 (127.27.75.193:41719)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:14:50.339002 13636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:54257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:14:50.353420 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42789
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:50.466048 13833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:50.466331 13833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:50.466374 13833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:50.470517 13833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:50.470588 13833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:50.470650 13833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:14:50.476330 13833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:50.477715 13833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:50.478999 13833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:50.489061 13839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:50.489065 13841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:50.489065 13838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:50.489356 13833 server_base.cc:1034] running on GCE node
I20260923 18:14:50.489910 13833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:50.490584 13833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:50.491760 13833 hybrid_clock.cc:648] HybridClock initialized: now 1790187290491737 us; error 37 us; skew 500 ppm
Sep 23 18:14:50 dist-test-slave-wsx5 krb5kdc[13591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:50.494693 13833 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:14:50.495638 13833 webserver.cc:460] Webserver started at http://127.27.75.194:38181/ using document root <none> and password file <none>
I20260923 18:14:50.496246 13833 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:50.496330 13833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:50.496538 13833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:50.498669 13833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "298dfef36ca342ce9aa93002123b8b9f"
format_stamp: "Formatted at 2026-09-23 18:14:50 on dist-test-slave-wsx5"
server_key: "3a1944a2d0b8f21d2da32fbd63ca53a8"
server_key_iv: "3700b4b60e3d792c382ef5c2d7721700"
server_key_version: "encryptionkey@0"
I20260923 18:14:50.499248 13833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "298dfef36ca342ce9aa93002123b8b9f"
format_stamp: "Formatted at 2026-09-23 18:14:50 on dist-test-slave-wsx5"
server_key: "3a1944a2d0b8f21d2da32fbd63ca53a8"
server_key_iv: "3700b4b60e3d792c382ef5c2d7721700"
server_key_version: "encryptionkey@0"
I20260923 18:14:50.503443 13833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:14:50.506103 13848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:50.507508 13833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:50.507637 13833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "298dfef36ca342ce9aa93002123b8b9f"
format_stamp: "Formatted at 2026-09-23 18:14:50 on dist-test-slave-wsx5"
server_key: "3a1944a2d0b8f21d2da32fbd63ca53a8"
server_key_iv: "3700b4b60e3d792c382ef5c2d7721700"
server_key_version: "encryptionkey@0"
I20260923 18:14:50.507750 13833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:50.551712 13833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:50.555409 13833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:50.555678 13833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:50.556514 13833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:50.557436 13833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:50.557487 13833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:50.557567 13833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:50.557592 13833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:50.566171 13833 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:37285
I20260923 18:14:50.566198 13962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:37285 every 8 connection(s)
I20260923 18:14:50.567266 13833 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:14:50.570786 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 13833
I20260923 18:14:50.570890 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:14:50.571166 27951 external_mini_cluster.cc:1428] Setting key 10336e88fa92d8370789059749e07982
Sep 23 18:14:50 dist-test-slave-wsx5 krb5kdc[13591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:14:50.581918 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.569244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:59339 (local address 127.27.75.254:42789)
0923 18:14:50.569372 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:50.569376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:50.570202 (+ 826us) server_negotiation.cc:407] Connection header received
0923 18:14:50.571652 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:50.571656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:50.571713 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:50.571797 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:50.573308 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.573833 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.575224 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.575464 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.575559 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.578305 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:50.578309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:50.578312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:50.578344 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:50.579674 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:50.580237 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:50.580240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:50.580241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:50.580325 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:50.580660 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:50.580664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:50.580665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:50.580876 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:50.580962 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:50.581586 (+ 624us) server_negotiation.cc:299] Negotiation successful
0923 18:14:50.581734 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:14:50.582801 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.569600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42789 (local address 127.27.75.194:59339)
0923 18:14:50.570053 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:50.570096 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:50.571404 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:50.571988 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:50.571998 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:50.572537 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:50.573158 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.573179 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.573968 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.573972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.575070 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.575083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.575629 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.575633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.575812 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.576386 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:50.576403 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:50.578091 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:50.579799 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:50.579805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:50.579825 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:50.580131 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:50.580407 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:50.580410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:50.580412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:50.580565 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:50.580975 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:50.580982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:50.581334 (+ 352us) client_negotiation.cc:769] Sending connection context
0923 18:14:50.581517 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 18:14:50.581886 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
I20260923 18:14:50.584041 13963 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42789
I20260923 18:14:50.584290 13963 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:50.584904 13963 heartbeater.cc:511] Master 127.27.75.254:42789 requested a full tablet report, sending...
I20260923 18:14:50.586194 13636 ts_manager.cc:194] Registered new tserver with Master: 298dfef36ca342ce9aa93002123b8b9f (127.27.75.194:37285)
I20260923 18:14:50.586807 13636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:59339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:14:50.601027 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42789
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:50.714455 13968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:50.714747 13968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:50.714802 13968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:50.718962 13968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:50.719033 13968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:50.719099 13968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:14:50.724882 13968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:50.726107 13968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:50.727169 13968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:50.737677 13976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:50.737772 13973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:50.737895 13968 server_base.cc:1034] running on GCE node
W20260923 18:14:50.737677 13974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:50.738425 13968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:50.739131 13968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:50.740341 13968 hybrid_clock.cc:648] HybridClock initialized: now 1790187290740310 us; error 45 us; skew 500 ppm
Sep 23 18:14:50 dist-test-slave-wsx5 krb5kdc[13591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:50.743333 13968 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:14:50.744310 13968 webserver.cc:460] Webserver started at http://127.27.75.195:35803/ using document root <none> and password file <none>
I20260923 18:14:50.744907 13968 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:50.744980 13968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:50.745204 13968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:50.747611 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf"
format_stamp: "Formatted at 2026-09-23 18:14:50 on dist-test-slave-wsx5"
server_key: "05c1f930d00d5765ece619dede3ac798"
server_key_iv: "2ad2cd3f496e235671c36d704f19d935"
server_key_version: "encryptionkey@0"
I20260923 18:14:50.748328 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf"
format_stamp: "Formatted at 2026-09-23 18:14:50 on dist-test-slave-wsx5"
server_key: "05c1f930d00d5765ece619dede3ac798"
server_key_iv: "2ad2cd3f496e235671c36d704f19d935"
server_key_version: "encryptionkey@0"
I20260923 18:14:50.752388 13968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:14:50.755188 13983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:50.756721 13968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:14:50.756839 13968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf"
format_stamp: "Formatted at 2026-09-23 18:14:50 on dist-test-slave-wsx5"
server_key: "05c1f930d00d5765ece619dede3ac798"
server_key_iv: "2ad2cd3f496e235671c36d704f19d935"
server_key_version: "encryptionkey@0"
I20260923 18:14:50.756945 13968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:50.773810 13968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:50.777022 13968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:50.777285 13968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:50.777936 13968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:50.778771 13968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:50.778816 13968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:50.778868 13968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:50.778888 13968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:50.787753 13968 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:41365
I20260923 18:14:50.787904 14097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:41365 every 8 connection(s)
I20260923 18:14:50.788688 13968 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:14:50.797909 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 13968
I20260923 18:14:50.798089 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
Sep 23 18:14:50 dist-test-slave-wsx5 krb5kdc[13591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:50.798414 27951 external_mini_cluster.cc:1428] Setting key 2febd31afa277d4fc6cc33f4f410edb2
I20260923 18:14:50.802480 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.790532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:47223 (local address 127.27.75.254:42789)
0923 18:14:50.790655 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:50.790658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:50.791524 (+ 866us) server_negotiation.cc:407] Connection header received
0923 18:14:50.792857 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:50.792861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:50.792930 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:50.793006 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:50.794216 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.794751 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.795879 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.796100 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.796186 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.798876 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:50.798880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:50.798882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:50.798914 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:50.800485 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:50.801007 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:50.801010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:50.801012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:50.801095 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:50.801417 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:50.801421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:50.801422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:50.801614 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:50.801699 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:50.802197 (+ 498us) server_negotiation.cc:299] Negotiation successful
0923 18:14:50.802320 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 18:14:50.803297 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.790887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42789 (local address 127.27.75.195:47223)
0923 18:14:50.791349 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:50.791417 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:50.792630 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:50.793219 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:50.793227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:50.793589 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:50.794073 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.794086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.794868 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.794872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.795754 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.795763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.796266 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.796271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.796433 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.796997 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:50.797013 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:50.798686 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:50.800650 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:50.800655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:50.800667 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:50.800914 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:50.801189 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:50.801193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:50.801195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:50.801334 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:50.801709 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:50.801716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:50.801973 (+ 257us) client_negotiation.cc:769] Sending connection context
0923 18:14:50.802139 (+ 166us) client_negotiation.cc:240] Negotiation successful
0923 18:14:50.802496 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":119,"threads_started":1}
I20260923 18:14:50.804360 14098 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42789
I20260923 18:14:50.804598 14098 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:50.805083 14098 heartbeater.cc:511] Master 127.27.75.254:42789 requested a full tablet report, sending...
I20260923 18:14:50.806142 13636 ts_manager.cc:194] Registered new tserver with Master: f3a2221d2ceb4adfaa5707ec23ddc0cf (127.27.75.195:41365)
I20260923 18:14:50.806861 13636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:47223
I20260923 18:14:50.812480 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:14:50.822032 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.814576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.27.75.254:42789)
0923 18:14:50.814714 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:50.814718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:50.814890 (+ 172us) server_negotiation.cc:407] Connection header received
0923 18:14:50.815002 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:50.815004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:50.815048 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:50.815117 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:50.815754 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.816270 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.817163 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.817390 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.817477 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.818997 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:50.819001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:50.819004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:50.819033 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:50.820242 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:50.820722 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:50.820726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:50.820728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:50.820801 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:50.821097 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:50.821100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:50.821102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:50.821373 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:50.821519 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:50.821784 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 18:14:50.821872 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 18:14:50.824800 13636 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:14:50.827904 13636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:14:50.842739 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.837009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.27.75.193:41719)
0923 18:14:50.837438 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:50.837445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:50.837812 (+ 367us) server_negotiation.cc:407] Connection header received
0923 18:14:50.838125 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:50.838131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:50.838372 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:50.838525 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:50.838823 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.839795 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.841213 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.842045 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.842154 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.842319 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:50.842481 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 18:14:50.842585 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":76,"threads_started":1}
I20260923 18:14:50.842728 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.836938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:41719 (local address 127.0.0.1:54150)
0923 18:14:50.837666 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:50.837718 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:50.837926 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:50.838519 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:50.838525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:50.838548 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:50.838698 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.838706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.839897 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.839900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.841035 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.841046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.842158 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.842162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.842283 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.842330 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 18:14:50.842436 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 18:14:50.842574 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":24,"thread_start_us":83,"threads_started":1}
I20260923 18:14:50.843560 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.837381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.27.75.194:37285)
0923 18:14:50.837821 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:50.837832 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:50.838354 (+ 522us) server_negotiation.cc:407] Connection header received
0923 18:14:50.838419 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:50.838422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:50.838603 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:50.838751 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:50.839146 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.840344 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.841872 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.842625 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.842728 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.842869 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:50.843317 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 18:14:50.843429 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":140,"threads_started":1}
I20260923 18:14:50.843545 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.836985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:37285 (local address 127.0.0.1:51906)
0923 18:14:50.838175 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:50.838187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:50.838262 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:50.838920 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:50.838922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:50.838939 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:50.839020 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.839026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.840493 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.840496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.841744 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.841754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.842935 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.842939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.843133 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.843185 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 18:14:50.843263 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 18:14:50.843374 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":140,"spinlock_wait_cycles":3200,"thread_start_us":152,"threads_started":1}
I20260923 18:14:50.843546 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.837701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:41365 (local address 127.0.0.1:49108)
0923 18:14:50.838162 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:50.838175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:50.838261 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:50.839184 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:50.839187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:50.839203 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:50.839277 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.839283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.840544 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.840548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.841605 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.841615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.842935 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.842939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.843147 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.843181 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 18:14:50.843263 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 18:14:50.843372 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":73,"spinlock_wait_cycles":768,"thread_start_us":80,"threads_started":1}
I20260923 18:14:50.844329 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.837883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.27.75.195:41365)
0923 18:14:50.838836 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:50.838847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:50.838868 (+ 21us) server_negotiation.cc:407] Connection header received
0923 18:14:50.838925 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:50.838928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:50.839057 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:50.839196 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:50.839369 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.840440 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.841754 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.842618 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.842740 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.842869 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:50.844098 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0923 18:14:50.844202 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":141,"threads_started":1}
I20260923 18:14:50.846513 13762 tablet_service.cc:1467] Processing CreateTablet for tablet ea8b7f1f21234a4da73e84b2dcedaced (DEFAULT_TABLE table=test-table [id=dc3eedd088dd4fbd98380a371c5f833c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:50.846997 14032 tablet_service.cc:1467] Processing CreateTablet for tablet ea8b7f1f21234a4da73e84b2dcedaced (DEFAULT_TABLE table=test-table [id=dc3eedd088dd4fbd98380a371c5f833c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:50.846858 13897 tablet_service.cc:1467] Processing CreateTablet for tablet ea8b7f1f21234a4da73e84b2dcedaced (DEFAULT_TABLE table=test-table [id=dc3eedd088dd4fbd98380a371c5f833c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:50.847558 13762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8b7f1f21234a4da73e84b2dcedaced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:50.847903 14032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8b7f1f21234a4da73e84b2dcedaced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:50.847903 13897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8b7f1f21234a4da73e84b2dcedaced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:50.854039 14115 tablet_bootstrap.cc:492] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6: Bootstrap starting.
I20260923 18:14:50.855404 14116 tablet_bootstrap.cc:492] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f: Bootstrap starting.
I20260923 18:14:50.855738 14117 tablet_bootstrap.cc:492] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: Bootstrap starting.
I20260923 18:14:50.857635 14117 tablet_bootstrap.cc:654] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:50.857635 14116 tablet_bootstrap.cc:654] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:50.857635 14115 tablet_bootstrap.cc:654] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:50.858633 14117 log.cc:824] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:50.858619 14115 log.cc:824] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:50.858800 14116 log.cc:824] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:50.860612 14117 tablet_bootstrap.cc:492] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: No bootstrap required, opened a new log
I20260923 18:14:50.860612 14116 tablet_bootstrap.cc:492] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f: No bootstrap required, opened a new log
I20260923 18:14:50.860622 14115 tablet_bootstrap.cc:492] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6: No bootstrap required, opened a new log
I20260923 18:14:50.860810 14116 ts_tablet_manager.cc:1397] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:14:50.860814 14117 ts_tablet_manager.cc:1397] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 18:14:50.860810 14115 ts_tablet_manager.cc:1397] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 18:14:50.864473 14115 raft_consensus.cc:348] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0158e9c589704e30bd90c6d1072d38e6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41719 } } peers { permanent_uuid: "298dfef36ca342ce9aa93002123b8b9f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 37285 } } peers { permanent_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41365 } }
I20260923 18:14:50.864763 14115 raft_consensus.cc:374] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:50.864821 14115 raft_consensus.cc:729] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0158e9c589704e30bd90c6d1072d38e6, State: Initialized, Role: FOLLOWER
I20260923 18:14:50.864849 14116 raft_consensus.cc:348] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0158e9c589704e30bd90c6d1072d38e6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41719 } } peers { permanent_uuid: "298dfef36ca342ce9aa93002123b8b9f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 37285 } } peers { permanent_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41365 } }
I20260923 18:14:50.865141 14116 raft_consensus.cc:374] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:50.864849 14117 raft_consensus.cc:348] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0158e9c589704e30bd90c6d1072d38e6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41719 } } peers { permanent_uuid: "298dfef36ca342ce9aa93002123b8b9f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 37285 } } peers { permanent_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41365 } }
I20260923 18:14:50.865203 14116 raft_consensus.cc:729] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298dfef36ca342ce9aa93002123b8b9f, State: Initialized, Role: FOLLOWER
I20260923 18:14:50.865212 14117 raft_consensus.cc:374] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:50.865254 14117 raft_consensus.cc:729] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a2221d2ceb4adfaa5707ec23ddc0cf, State: Initialized, Role: FOLLOWER
I20260923 18:14:50.865351 14115 consensus_queue.cc:260] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0158e9c589704e30bd90c6d1072d38e6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41719 } } peers { permanent_uuid: "298dfef36ca342ce9aa93002123b8b9f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 37285 } } peers { permanent_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41365 } }
I20260923 18:14:50.865638 14116 consensus_queue.cc:260] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0158e9c589704e30bd90c6d1072d38e6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41719 } } peers { permanent_uuid: "298dfef36ca342ce9aa93002123b8b9f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 37285 } } peers { permanent_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41365 } }
I20260923 18:14:50.865643 14117 consensus_queue.cc:260] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0158e9c589704e30bd90c6d1072d38e6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41719 } } peers { permanent_uuid: "298dfef36ca342ce9aa93002123b8b9f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 37285 } } peers { permanent_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41365 } }
I20260923 18:14:50.866144 13828 heartbeater.cc:503] Master 127.27.75.254:42789 was elected leader, sending a full tablet report...
I20260923 18:14:50.866461 14115 ts_tablet_manager.cc:1428] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 18:14:50.866607 14116 ts_tablet_manager.cc:1428] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:14:50.866711 14117 ts_tablet_manager.cc:1428] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 18:14:50.866969 13963 heartbeater.cc:503] Master 127.27.75.254:42789 was elected leader, sending a full tablet report...
I20260923 18:14:50.866992 14098 heartbeater.cc:503] Master 127.27.75.254:42789 was elected leader, sending a full tablet report...
I20260923 18:14:50.989276 14122 raft_consensus.cc:482] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:50.989480 14122 raft_consensus.cc:504] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0158e9c589704e30bd90c6d1072d38e6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41719 } } peers { permanent_uuid: "298dfef36ca342ce9aa93002123b8b9f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 37285 } } peers { permanent_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41365 } }
I20260923 18:14:50.990670 14122 leader_election.cc:288] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0158e9c589704e30bd90c6d1072d38e6 (127.27.75.193:41719), f3a2221d2ceb4adfaa5707ec23ddc0cf (127.27.75.195:41365)
I20260923 18:14:50.994244 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.991190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:36823 (local address 127.27.75.193:41719)
0923 18:14:50.991303 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:50.991307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:50.991322 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:14:50.991374 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:50.991377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:50.991421 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:50.991482 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:50.991750 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.992246 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.993321 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.993730 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.993842 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.993895 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:50.994058 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 18:14:50.994129 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:14:50.994244 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.990877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:41719 (local address 127.27.75.194:36823)
0923 18:14:50.991175 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:50.991190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:50.991276 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:50.991511 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:50.991514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:50.991532 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:50.991639 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.991643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.992357 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.992361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.993204 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.993213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.993839 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.993843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.993948 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.993965 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 18:14:50.994028 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 18:14:50.994105 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":1}
I20260923 18:14:50.994514 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.991343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:47685 (local address 127.27.75.195:41365)
0923 18:14:50.991461 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:50.991466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:50.991583 (+ 117us) server_negotiation.cc:407] Connection header received
0923 18:14:50.991691 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:50.991694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:50.991745 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:50.991804 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:50.992070 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.992552 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.993583 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.993969 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.994074 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.994125 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:50.994301 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 18:14:50.994397 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 18:14:50.994840 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:50.990935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:41365 (local address 127.27.75.194:47685)
0923 18:14:50.991495 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:50.991508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:50.991606 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:50.991827 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:50.991830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:50.991846 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:50.991968 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.991973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.992660 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.992663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.993465 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:50.993474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:50.994091 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:50.994095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:50.994188 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:50.994209 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 18:14:50.994279 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 18:14:50.994716 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":122,"threads_started":1}
I20260923 18:14:50.994959 13782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8b7f1f21234a4da73e84b2dcedaced" candidate_uuid: "298dfef36ca342ce9aa93002123b8b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0158e9c589704e30bd90c6d1072d38e6" is_pre_election: true
I20260923 18:14:50.995296 13782 raft_consensus.cc:2454] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 298dfef36ca342ce9aa93002123b8b9f in term 0.
I20260923 18:14:50.995476 14052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8b7f1f21234a4da73e84b2dcedaced" candidate_uuid: "298dfef36ca342ce9aa93002123b8b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" is_pre_election: true
I20260923 18:14:50.995770 14052 raft_consensus.cc:2454] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 298dfef36ca342ce9aa93002123b8b9f in term 0.
I20260923 18:14:50.995842 13852 leader_election.cc:302] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0158e9c589704e30bd90c6d1072d38e6, 298dfef36ca342ce9aa93002123b8b9f; no voters:
I20260923 18:14:50.996156 14122 raft_consensus.cc:2781] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:14:50.996258 14122 raft_consensus.cc:482] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:14:50.996294 14122 raft_consensus.cc:3037] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:50.997579 14122 raft_consensus.cc:504] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0158e9c589704e30bd90c6d1072d38e6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41719 } } peers { permanent_uuid: "298dfef36ca342ce9aa93002123b8b9f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 37285 } } peers { permanent_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41365 } }
I20260923 18:14:50.997967 14122 leader_election.cc:288] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [CANDIDATE]: Term 1 election: Requested vote from peers 0158e9c589704e30bd90c6d1072d38e6 (127.27.75.193:41719), f3a2221d2ceb4adfaa5707ec23ddc0cf (127.27.75.195:41365)
I20260923 18:14:50.998447 13782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8b7f1f21234a4da73e84b2dcedaced" candidate_uuid: "298dfef36ca342ce9aa93002123b8b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0158e9c589704e30bd90c6d1072d38e6"
I20260923 18:14:50.998546 14052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8b7f1f21234a4da73e84b2dcedaced" candidate_uuid: "298dfef36ca342ce9aa93002123b8b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf"
I20260923 18:14:50.998593 13782 raft_consensus.cc:3037] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:50.998672 14052 raft_consensus.cc:3037] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:50.999938 13782 raft_consensus.cc:2454] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 298dfef36ca342ce9aa93002123b8b9f in term 1.
I20260923 18:14:50.999938 14052 raft_consensus.cc:2454] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 298dfef36ca342ce9aa93002123b8b9f in term 1.
I20260923 18:14:51.000303 13851 leader_election.cc:302] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 298dfef36ca342ce9aa93002123b8b9f, f3a2221d2ceb4adfaa5707ec23ddc0cf; no voters:
I20260923 18:14:51.000490 14122 raft_consensus.cc:2781] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:51.000741 14122 raft_consensus.cc:686] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 1 LEADER]: Becoming Leader. State: Replica: 298dfef36ca342ce9aa93002123b8b9f, State: Running, Role: LEADER
I20260923 18:14:51.001098 14122 consensus_queue.cc:237] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0158e9c589704e30bd90c6d1072d38e6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41719 } } peers { permanent_uuid: "298dfef36ca342ce9aa93002123b8b9f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 37285 } } peers { permanent_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41365 } }
I20260923 18:14:51.004768 13636 catalog_manager.cc:5495] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 298dfef36ca342ce9aa93002123b8b9f (127.27.75.194). New cstate: current_term: 1 leader_uuid: "298dfef36ca342ce9aa93002123b8b9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0158e9c589704e30bd90c6d1072d38e6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "298dfef36ca342ce9aa93002123b8b9f" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 37285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41365 } health_report { overall_health: UNKNOWN } } }
I20260923 18:14:51.037266 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.033588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51908 (local address 127.27.75.194:37285)
0923 18:14:51.033715 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:51.033718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:51.033732 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:14:51.033866 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:51.033869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:51.033935 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:51.034028 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:14:51.034289 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.034868 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.035805 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.036049 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.036134 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:51.036373 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:14:51.036795 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:14:51.036916 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:51.037052 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 18:14:51.037090 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260923 18:14:51.039690 14099 tablet.cc:2369] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:14:51.043650 14052 raft_consensus.cc:1261] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf [term 1 FOLLOWER]: Refusing update from remote peer 298dfef36ca342ce9aa93002123b8b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:51.043650 13782 raft_consensus.cc:1261] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6 [term 1 FOLLOWER]: Refusing update from remote peer 298dfef36ca342ce9aa93002123b8b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:51.044353 14122 consensus_queue.cc:1035] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0158e9c589704e30bd90c6d1072d38e6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 41719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:51.044616 14125 consensus_queue.cc:1035] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3a2221d2ceb4adfaa5707ec23ddc0cf" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 41365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:51.050916 14128 mvcc.cc:205] Tried to move back new op lower bound from 7332607144106831872 to 7332607143943782400. Current Snapshot: MvccSnapshot[applied={T|T < 7332607144106831872}]
I20260923 18:14:51.051635 14130 mvcc.cc:205] Tried to move back new op lower bound from 7332607144106831872 to 7332607143943782400. Current Snapshot: MvccSnapshot[applied={T|T < 7332607144106831872}]
I20260923 18:14:51.053927 14129 mvcc.cc:205] Tried to move back new op lower bound from 7332607144106831872 to 7332607143943782400. Current Snapshot: MvccSnapshot[applied={T|T < 7332607144106831872}]
I20260923 18:14:51.061441 13634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:51.061633 13634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:51.064415 13634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c387f31b45674788bfdaf29779e480c7: Sending DeleteTablet for 3 replicas of tablet ea8b7f1f21234a4da73e84b2dcedaced
I20260923 18:14:51.065474 14032 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8b7f1f21234a4da73e84b2dcedaced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:51 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:49108
I20260923 18:14:51.065526 13762 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8b7f1f21234a4da73e84b2dcedaced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:51 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:54150
I20260923 18:14:51.066073 14138 tablet_replica.cc:330] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: stopping tablet replica
I20260923 18:14:51.066275 13897 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8b7f1f21234a4da73e84b2dcedaced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:51 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:51906
I20260923 18:14:51.066334 14137 tablet_replica.cc:330] T ea8b7f1f21234a4da73e84b2dcedaced P 0158e9c589704e30bd90c6d1072d38e6: stopping tablet replica
I20260923 18:14:51.066847 14139 tablet_replica.cc:330] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f: stopping tablet replica
I20260923 18:14:51.066907 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 13698
I20260923 18:14:51.067085 14138 raft_consensus.cc:2229] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:51.067281 14139 raft_consensus.cc:2229] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:14:51.067456 14138 raft_consensus.cc:2258] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:51.067931 14139 raft_consensus.cc:2258] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:51.069231 14138 ts_tablet_manager.cc:1905] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:51.069542 14139 ts_tablet_manager.cc:1905] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:51.071986 14138 ts_tablet_manager.cc:1918] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:51.072093 14138 log.cc:1196] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/ea8b7f1f21234a4da73e84b2dcedaced
I20260923 18:14:51.072394 14138 ts_tablet_manager.cc:1939] T ea8b7f1f21234a4da73e84b2dcedaced P f3a2221d2ceb4adfaa5707ec23ddc0cf: Deleting consensus metadata
I20260923 18:14:51.072387 14139 ts_tablet_manager.cc:1918] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:51.072597 14139 log.cc:1196] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/ea8b7f1f21234a4da73e84b2dcedaced
I20260923 18:14:51.073009 14139 ts_tablet_manager.cc:1939] T ea8b7f1f21234a4da73e84b2dcedaced P 298dfef36ca342ce9aa93002123b8b9f: Deleting consensus metadata
I20260923 18:14:51.073807 13622 catalog_manager.cc:4841] TS f3a2221d2ceb4adfaa5707ec23ddc0cf (127.27.75.195:41365): tablet ea8b7f1f21234a4da73e84b2dcedaced (table test-table [id=dc3eedd088dd4fbd98380a371c5f833c]) successfully deleted
I20260923 18:14:51.074239 13622 catalog_manager.cc:4841] TS 298dfef36ca342ce9aa93002123b8b9f (127.27.75.194:37285): tablet ea8b7f1f21234a4da73e84b2dcedaced (table test-table [id=dc3eedd088dd4fbd98380a371c5f833c]) successfully deleted
W20260923 18:14:51.076915 13623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:41719 (error 108)
I20260923 18:14:51.077090 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 13833
I20260923 18:14:51.078295 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.077985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:41719 (local address 127.0.0.1:54160)
0923 18:14:51.078134 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:51.078204 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 18:14:51.078500 13623 catalog_manager.cc:4584] TS 0158e9c589704e30bd90c6d1072d38e6 (127.27.75.193:41719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea8b7f1f21234a4da73e84b2dcedaced: Network error: Client connection negotiation failed: client connection to 127.27.75.193:41719: connect: Connection refused (error 111)
I20260923 18:14:51.082660 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 13968
I20260923 18:14:51.087981 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 13604
2026-09-23T18:14:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2374 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T18:14:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:14:51Z Disabled control of system clock
I20260923 18:14:51.116734 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:45505
--webserver_interface=127.27.75.254
--webserver_port=0
--builtin_ntp_servers=127.27.75.212:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:45505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:14:51.227049 14144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:51.227344 14144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:51.227386 14144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:51.231500 14144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:14:51.231591 14144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:51.231608 14144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:51.231624 14144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:14:51.231638 14144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:14:51.236605 14144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:45505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:45505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:51.237691 14144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:51.238998 14144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:51.247673 14150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:51.247634 14149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:51.247654 14152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:51.247728 14144 server_base.cc:1034] running on GCE node
I20260923 18:14:51.248418 14144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:51.249625 14144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:51.250876 14144 hybrid_clock.cc:648] HybridClock initialized: now 1790187291250853 us; error 33 us; skew 500 ppm
I20260923 18:14:51.252761 14144 webserver.cc:460] Webserver started at http://127.27.75.254:43197/ using document root <none> and password file <none>
I20260923 18:14:51.253368 14144 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:51.253440 14144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:51.253643 14144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:51.255676 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "e980392825874f76a134a435f86abe20"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "5f5d5de4867e25796811eb86aac0c9ff"
server_key_iv: "1746457590b399700553191d5c6c04e0"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.256218 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "e980392825874f76a134a435f86abe20"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "5f5d5de4867e25796811eb86aac0c9ff"
server_key_iv: "1746457590b399700553191d5c6c04e0"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.259964 14144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 18:14:51.262650 14158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:51.263926 14144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:14:51.264056 14144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "e980392825874f76a134a435f86abe20"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "5f5d5de4867e25796811eb86aac0c9ff"
server_key_iv: "1746457590b399700553191d5c6c04e0"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.264161 14144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:51.281358 14144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:51.282073 14144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:51.282246 14144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:51.289208 14144 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:45505
I20260923 18:14:51.289222 14210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:45505 every 8 connection(s)
I20260923 18:14:51.290185 14144 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:14:51.292687 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 14144
I20260923 18:14:51.292856 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:14:51.293157 27951 external_mini_cluster.cc:1428] Setting key 757777ceac540f53423bc1ac80eae3d5
I20260923 18:14:51.293520 14211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:51.300154 14211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20: Bootstrap starting.
I20260923 18:14:51.302511 14211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:51.302563 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.294607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.27.75.254:45505)
0923 18:14:51.295284 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:51.295328 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:51.295366 (+ 38us) server_negotiation.cc:407] Connection header received
0923 18:14:51.296244 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:51.296263 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:51.296711 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:51.297041 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:51.297513 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.298483 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.299358 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.299831 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.299976 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:51.300824 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:51.300830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:51.300843 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:51.301281 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:51.301345 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:51.301367 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:51.301455 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:51.301600 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 18:14:51.301858 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":152,"threads_started":1}
I20260923 18:14:51.303401 14211 log.cc:824] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:51.305411 14211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20: No bootstrap required, opened a new log
I20260923 18:14:51.308979 14211 raft_consensus.cc:348] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e980392825874f76a134a435f86abe20" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 45505 } }
I20260923 18:14:51.309268 14211 raft_consensus.cc:374] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:51.309334 14211 raft_consensus.cc:729] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e980392825874f76a134a435f86abe20, State: Initialized, Role: FOLLOWER
I20260923 18:14:51.309724 14211 consensus_queue.cc:260] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e980392825874f76a134a435f86abe20" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 45505 } }
I20260923 18:14:51.309847 14211 raft_consensus.cc:388] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:14:51.309887 14211 raft_consensus.cc:482] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:14:51.309965 14211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:51.310875 14211 raft_consensus.cc:504] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e980392825874f76a134a435f86abe20" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 45505 } }
I20260923 18:14:51.311203 14211 leader_election.cc:302] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e980392825874f76a134a435f86abe20; no voters:
I20260923 18:14:51.311481 14211 leader_election.cc:288] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:14:51.311600 14216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:51.311815 14216 raft_consensus.cc:686] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [term 1 LEADER]: Becoming Leader. State: Replica: e980392825874f76a134a435f86abe20, State: Running, Role: LEADER
I20260923 18:14:51.312129 14216 consensus_queue.cc:237] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e980392825874f76a134a435f86abe20" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 45505 } }
I20260923 18:14:51.312692 14211 sys_catalog.cc:564] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:14:51.313642 14218 sys_catalog.cc:455] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e980392825874f76a134a435f86abe20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e980392825874f76a134a435f86abe20" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 45505 } } }
I20260923 18:14:51.313767 14218 sys_catalog.cc:458] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:51.314137 14219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e980392825874f76a134a435f86abe20. Latest consensus state: current_term: 1 leader_uuid: "e980392825874f76a134a435f86abe20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e980392825874f76a134a435f86abe20" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 45505 } } }
I20260923 18:14:51.314227 14219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20 [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:51.314431 14226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:14:51.317726 14226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:14:51.323279 14226 catalog_manager.cc:1347] Generated new cluster ID: 5a5da554e80e4ebeab7ad150099ab52d
I20260923 18:14:51.323364 14226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:14:51.342239 14226 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:14:51.343266 14226 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:14:51.355894 14226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20: Generated new TSK 0
I20260923 18:14:51.356645 14226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:14:51.365799 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:45505
--builtin_ntp_servers=127.27.75.212:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:14:51.477950 14236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:51.478251 14236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:51.478309 14236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:51.482432 14236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:51.482498 14236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:51.482579 14236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:14:51.487550 14236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:45505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:51.488688 14236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:51.490139 14236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:51.499922 14242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:51.499905 14241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:51.500033 14236 server_base.cc:1034] running on GCE node
W20260923 18:14:51.499905 14244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:51.500551 14236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:51.501252 14236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:51.502417 14236 hybrid_clock.cc:648] HybridClock initialized: now 1790187291502395 us; error 33 us; skew 500 ppm
I20260923 18:14:51.504236 14236 webserver.cc:460] Webserver started at http://127.27.75.193:41317/ using document root <none> and password file <none>
I20260923 18:14:51.504842 14236 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:51.504922 14236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:51.505152 14236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:51.507241 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "e003850825e44bc6a5e020118b3521c6"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "1c399db0b9cb9e23d4a30c37d524aedb"
server_key_iv: "78f6ab0b71359f39f62c91d08c45b6ca"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.507807 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "e003850825e44bc6a5e020118b3521c6"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "1c399db0b9cb9e23d4a30c37d524aedb"
server_key_iv: "78f6ab0b71359f39f62c91d08c45b6ca"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.511641 14236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:14:51.514305 14250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:51.515713 14236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:14:51.515846 14236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "e003850825e44bc6a5e020118b3521c6"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "1c399db0b9cb9e23d4a30c37d524aedb"
server_key_iv: "78f6ab0b71359f39f62c91d08c45b6ca"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.515974 14236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:51.550251 14236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:51.551106 14236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:51.551298 14236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:51.551930 14236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:51.552953 14236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:51.553001 14236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:51.553056 14236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:51.553083 14236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:51.562171 14236 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36253
I20260923 18:14:51.562198 14364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36253 every 8 connection(s)
I20260923 18:14:51.563169 14236 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:14:51.573356 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 14236
I20260923 18:14:51.573433 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.565197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:35695 (local address 127.27.75.254:45505)
0923 18:14:51.565361 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:51.565365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:51.566467 (+ 1102us) server_negotiation.cc:407] Connection header received
0923 18:14:51.568002 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:51.568004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:51.568056 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:51.568133 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:51.569306 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.569860 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.571162 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.571395 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.571490 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:51.572693 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:51.572696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:51.572699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:51.572726 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:51.572746 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:51.572755 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:51.572952 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:51.573168 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 18:14:51.573271 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:14:51.573491 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:14:51.573772 27951 external_mini_cluster.cc:1428] Setting key 3613b79a93e1b409fe89261dff0e84f1
I20260923 18:14:51.574530 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.565584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:45505 (local address 127.27.75.193:35695)
0923 18:14:51.566310 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:51.566353 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:51.567751 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:51.568375 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:51.568383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:51.568493 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:51.569082 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:51.569097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.570003 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.570006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:51.571005 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:51.571017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.571571 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.571575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:51.571738 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:51.572370 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:14:51.572371 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:51.572415 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:14:51.572494 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:51.572952 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:51.572961 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:51.572966 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 18:14:51.573095 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 18:14:51.573425 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":118,"threads_started":1}
I20260923 18:14:51.575753 14365 heartbeater.cc:347] Connected to a master server at 127.27.75.254:45505
I20260923 18:14:51.576186 14365 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:51.575912 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:45505
--builtin_ntp_servers=127.27.75.212:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:14:51.576812 14365 heartbeater.cc:511] Master 127.27.75.254:45505 requested a full tablet report, sending...
I20260923 18:14:51.578389 14175 ts_manager.cc:194] Registered new tserver with Master: e003850825e44bc6a5e020118b3521c6 (127.27.75.193:36253)
I20260923 18:14:51.579533 14175 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.193:35695
W20260923 18:14:51.688541 14368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:51.688828 14368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:51.688874 14368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:51.693202 14368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:51.693287 14368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:51.693348 14368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:14:51.698730 14368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:45505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:51.699944 14368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:51.701475 14368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:51.711416 14376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:51.711503 14373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:51.711416 14374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:51.711664 14368 server_base.cc:1034] running on GCE node
I20260923 18:14:51.712262 14368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:51.712996 14368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:51.714185 14368 hybrid_clock.cc:648] HybridClock initialized: now 1790187291714137 us; error 66 us; skew 500 ppm
I20260923 18:14:51.716208 14368 webserver.cc:460] Webserver started at http://127.27.75.194:45913/ using document root <none> and password file <none>
I20260923 18:14:51.716805 14368 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:51.716885 14368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:51.717096 14368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:51.719272 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "cbee31702a2945b1be8ef6318de80399"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "5e718f589baecb5713c9e4e4294b0f0b"
server_key_iv: "f8ec7e77dd0abd764f7d3036dac161eb"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.719883 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "cbee31702a2945b1be8ef6318de80399"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "5e718f589baecb5713c9e4e4294b0f0b"
server_key_iv: "f8ec7e77dd0abd764f7d3036dac161eb"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.723817 14368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:14:51.729173 14382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:51.730893 14368 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260923 18:14:51.731053 14368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "cbee31702a2945b1be8ef6318de80399"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "5e718f589baecb5713c9e4e4294b0f0b"
server_key_iv: "f8ec7e77dd0abd764f7d3036dac161eb"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.731204 14368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:51.749758 14368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:51.750474 14368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:51.750641 14368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:51.751214 14368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:51.752177 14368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:51.752224 14368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:51.752279 14368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:51.752313 14368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:51.760807 14368 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:43443
I20260923 18:14:51.760844 14496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:43443 every 8 connection(s)
I20260923 18:14:51.761775 14368 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:14:51.764966 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 14368
I20260923 18:14:51.765085 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:14:51.765398 27951 external_mini_cluster.cc:1428] Setting key 745ba572b184e17d39e3cece03612521
I20260923 18:14:51.767393 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:45505
--builtin_ntp_servers=127.27.75.212:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:14:51.772063 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.763663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:33671 (local address 127.27.75.254:45505)
0923 18:14:51.763839 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:51.763843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:51.764926 (+ 1083us) server_negotiation.cc:407] Connection header received
0923 18:14:51.766506 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:51.766509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:51.766556 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:51.766653 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:51.767770 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.768478 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.769919 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.770151 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.770244 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:51.771444 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:51.771447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:51.771450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:51.771466 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:51.771489 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:51.771497 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:51.771558 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:51.771710 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 18:14:51.771784 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 18:14:51.773022 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.764078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:45505 (local address 127.27.75.194:33671)
0923 18:14:51.764782 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:51.764825 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:51.766284 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:51.766913 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:51.766922 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:51.767028 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:51.767606 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:51.767627 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.768643 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.768647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:51.769789 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:51.769801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.770301 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.770305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:51.770494 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:51.771157 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:14:51.771159 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:51.771204 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:14:51.771277 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:51.771572 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:51.771581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:51.771587 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:14:51.771702 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 18:14:51.772086 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":113,"threads_started":1}
I20260923 18:14:51.774259 14497 heartbeater.cc:347] Connected to a master server at 127.27.75.254:45505
I20260923 18:14:51.774498 14497 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:51.775035 14497 heartbeater.cc:511] Master 127.27.75.254:45505 requested a full tablet report, sending...
I20260923 18:14:51.776125 14175 ts_manager.cc:194] Registered new tserver with Master: cbee31702a2945b1be8ef6318de80399 (127.27.75.194:43443)
I20260923 18:14:51.776925 14175 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.194:33671
W20260923 18:14:51.881274 14501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:51.881560 14501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:51.881597 14501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:51.885718 14501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:51.885790 14501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:51.885848 14501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:14:51.890841 14501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:45505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:51.891934 14501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:51.893369 14501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:51.903173 14507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:51.903249 14506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:51.903312 14501 server_base.cc:1034] running on GCE node
W20260923 18:14:51.903204 14509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:51.903939 14501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:51.904610 14501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:51.905773 14501 hybrid_clock.cc:648] HybridClock initialized: now 1790187291905756 us; error 29 us; skew 500 ppm
I20260923 18:14:51.907864 14501 webserver.cc:460] Webserver started at http://127.27.75.195:35039/ using document root <none> and password file <none>
I20260923 18:14:51.908488 14501 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:51.908555 14501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:51.908726 14501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:51.910919 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "17acbd4b957141008f6c40974f9dfe95"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "02f77977eead8917e0a548e8fe32c9c6"
server_key_iv: "d8567a7d664337a55f1d632ec41b9a44"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.911489 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "17acbd4b957141008f6c40974f9dfe95"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "02f77977eead8917e0a548e8fe32c9c6"
server_key_iv: "d8567a7d664337a55f1d632ec41b9a44"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.915592 14501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 18:14:51.918165 14515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:51.919452 14501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:14:51.919564 14501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "17acbd4b957141008f6c40974f9dfe95"
format_stamp: "Formatted at 2026-09-23 18:14:51 on dist-test-slave-wsx5"
server_key: "02f77977eead8917e0a548e8fe32c9c6"
server_key_iv: "d8567a7d664337a55f1d632ec41b9a44"
server_key_version: "encryptionkey@0"
I20260923 18:14:51.919674 14501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:51.949258 14501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:51.950002 14501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:51.950192 14501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:51.950800 14501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:51.951784 14501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:51.951833 14501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:51.951874 14501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:51.951890 14501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:51.960791 14501 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:40601
I20260923 18:14:51.960853 14628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:40601 every 8 connection(s)
I20260923 18:14:51.961797 14501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:14:51.964188 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 14501
I20260923 18:14:51.964282 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:14:51.964557 27951 external_mini_cluster.cc:1428] Setting key 28dd535dc487a33dca8f62c2d418e3ec
I20260923 18:14:51.972031 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.963683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:51425 (local address 127.27.75.254:45505)
0923 18:14:51.963830 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:51.963834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:51.965009 (+ 1175us) server_negotiation.cc:407] Connection header received
0923 18:14:51.966657 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:51.966660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:51.966719 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:51.966802 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:51.967862 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.968484 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.969937 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.970162 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.970267 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:51.971445 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:51.971448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:51.971450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:51.971466 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:51.971489 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:51.971498 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:51.971564 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:51.971795 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 18:14:51.971863 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 18:14:51.972982 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.964112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:45505 (local address 127.27.75.195:51425)
0923 18:14:51.964831 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:51.964885 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:51.966336 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:51.967022 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:51.967029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:51.967138 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:51.967688 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:51.967707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.968676 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.968679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:51.969774 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:51.969785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.970353 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.970356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:51.970515 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:51.971129 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:14:51.971131 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:51.971175 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:14:51.971249 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:51.971601 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:51.971609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:51.971615 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:14:51.971757 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 18:14:51.972055 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":128,"threads_started":1}
I20260923 18:14:51.974210 14629 heartbeater.cc:347] Connected to a master server at 127.27.75.254:45505
I20260923 18:14:51.974472 14629 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:51.975005 14629 heartbeater.cc:511] Master 127.27.75.254:45505 requested a full tablet report, sending...
I20260923 18:14:51.976130 14175 ts_manager.cc:194] Registered new tserver with Master: 17acbd4b957141008f6c40974f9dfe95 (127.27.75.195:40601)
I20260923 18:14:51.976732 14175 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.195:51425
I20260923 18:14:51.978703 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:14:51.984160 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.980632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.27.75.254:45505)
0923 18:14:51.980773 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:51.980778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:51.980996 (+ 218us) server_negotiation.cc:407] Connection header received
0923 18:14:51.981057 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:51.981059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:51.981099 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:51.981195 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:51.981415 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.981908 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.982711 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:51.982912 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:51.983010 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:51.983676 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:51.983678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:51.983681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:51.983692 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:51.983711 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:51.983719 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:51.983817 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:51.983945 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 18:14:51.984015 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 18:14:51.986980 14175 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:14:51.990492 14175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:14:52.005203 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.998999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:40601 (local address 127.0.0.1:40758)
0923 18:14:51.999487 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:51.999536 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:51.999661 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:52.000646 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:52.000652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:52.000677 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:52.000854 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:52.000862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.002327 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.002330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:52.003516 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:52.003525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.004582 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.004586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:52.004710 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:52.004750 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 18:14:52.004861 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 18:14:52.004999 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":22,"thread_start_us":91,"threads_started":1}
I20260923 18:14:52.005381 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.999882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55002 (local address 127.27.75.194:43443)
0923 18:14:52.000313 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:52.000325 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:52.001002 (+ 677us) server_negotiation.cc:407] Connection header received
0923 18:14:52.001219 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:52.001223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:52.001368 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:52.001491 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:52.001696 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.002503 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.003948 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.004690 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.004785 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:52.004943 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:52.005084 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 18:14:52.005214 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":125,"threads_started":1}
I20260923 18:14:52.005489 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.999763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.27.75.193:36253)
0923 18:14:52.000539 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:52.000550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:52.000568 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:14:52.000643 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:52.000648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:52.000832 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:52.001268 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:52.001404 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.002433 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.003920 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.004485 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.004585 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:52.004776 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:52.005230 (+ 454us) server_negotiation.cc:299] Negotiation successful
0923 18:14:52.005339 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":102,"threads_started":1}
I20260923 18:14:52.005698 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.999701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:43443 (local address 127.0.0.1:55002)
0923 18:14:52.000903 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:52.000916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:52.001067 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:52.001493 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:52.001496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:52.001513 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:52.001597 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:52.001604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.002606 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.002609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:52.003784 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:52.003790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.004844 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.004859 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:52.004951 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:52.004975 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:14:52.005046 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 18:14:52.005148 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"mutex_wait_us":47,"thread_start_us":92,"threads_started":1}
I20260923 18:14:52.005847 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.999188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36253 (local address 127.0.0.1:36782)
0923 18:14:51.999905 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:51.999916 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:51.999982 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:52.000986 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:52.000989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:52.001007 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:52.001100 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:52.001106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.002544 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.002547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:52.003779 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:52.003790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.004838 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.004841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:52.004920 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:52.004942 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:14:52.005015 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 18:14:52.005072 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260923 18:14:52.006008 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:51.999157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.27.75.195:40601)
0923 18:14:52.000194 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:52.000205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:52.000228 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:14:52.000304 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:52.000310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:52.000490 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:52.000645 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:52.001001 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.002201 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.003651 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.004443 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.004556 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:52.004765 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:52.005723 (+ 958us) server_negotiation.cc:299] Negotiation successful
0923 18:14:52.005840 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":155,"threads_started":1}
I20260923 18:14:52.008610 14563 tablet_service.cc:1467] Processing CreateTablet for tablet 20adebb204bc44d482a1f10f06e8f71f (DEFAULT_TABLE table=test-table [id=9f1ac7d776404d42aadf605b1039ad4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:52.008685 14430 tablet_service.cc:1467] Processing CreateTablet for tablet 20adebb204bc44d482a1f10f06e8f71f (DEFAULT_TABLE table=test-table [id=9f1ac7d776404d42aadf605b1039ad4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:52.008685 14298 tablet_service.cc:1467] Processing CreateTablet for tablet 20adebb204bc44d482a1f10f06e8f71f (DEFAULT_TABLE table=test-table [id=9f1ac7d776404d42aadf605b1039ad4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:52.009626 14563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20adebb204bc44d482a1f10f06e8f71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:52.009740 14298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20adebb204bc44d482a1f10f06e8f71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:52.009795 14430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20adebb204bc44d482a1f10f06e8f71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:52.016824 14646 tablet_bootstrap.cc:492] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95: Bootstrap starting.
I20260923 18:14:52.017503 14647 tablet_bootstrap.cc:492] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6: Bootstrap starting.
I20260923 18:14:52.017585 14648 tablet_bootstrap.cc:492] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399: Bootstrap starting.
I20260923 18:14:52.019821 14647 tablet_bootstrap.cc:654] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:52.019821 14648 tablet_bootstrap.cc:654] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:52.020709 14647 log.cc:824] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:52.020709 14648 log.cc:824] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:52.021349 14646 tablet_bootstrap.cc:654] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:52.022162 14646 log.cc:824] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:52.023020 14648 tablet_bootstrap.cc:492] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399: No bootstrap required, opened a new log
I20260923 18:14:52.023020 14647 tablet_bootstrap.cc:492] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6: No bootstrap required, opened a new log
I20260923 18:14:52.023217 14647 ts_tablet_manager.cc:1397] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 18:14:52.023217 14648 ts_tablet_manager.cc:1397] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:14:52.024045 14646 tablet_bootstrap.cc:492] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95: No bootstrap required, opened a new log
I20260923 18:14:52.024230 14646 ts_tablet_manager.cc:1397] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260923 18:14:52.027380 14646 raft_consensus.cc:348] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17acbd4b957141008f6c40974f9dfe95" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 40601 } } peers { permanent_uuid: "e003850825e44bc6a5e020118b3521c6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36253 } } peers { permanent_uuid: "cbee31702a2945b1be8ef6318de80399" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43443 } }
I20260923 18:14:52.027630 14646 raft_consensus.cc:374] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:52.027683 14646 raft_consensus.cc:729] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17acbd4b957141008f6c40974f9dfe95, State: Initialized, Role: FOLLOWER
I20260923 18:14:52.027518 14648 raft_consensus.cc:348] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17acbd4b957141008f6c40974f9dfe95" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 40601 } } peers { permanent_uuid: "e003850825e44bc6a5e020118b3521c6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36253 } } peers { permanent_uuid: "cbee31702a2945b1be8ef6318de80399" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43443 } }
I20260923 18:14:52.027518 14647 raft_consensus.cc:348] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17acbd4b957141008f6c40974f9dfe95" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 40601 } } peers { permanent_uuid: "e003850825e44bc6a5e020118b3521c6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36253 } } peers { permanent_uuid: "cbee31702a2945b1be8ef6318de80399" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43443 } }
I20260923 18:14:52.027761 14648 raft_consensus.cc:374] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:52.027761 14647 raft_consensus.cc:374] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:52.027818 14647 raft_consensus.cc:729] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e003850825e44bc6a5e020118b3521c6, State: Initialized, Role: FOLLOWER
I20260923 18:14:52.027828 14648 raft_consensus.cc:729] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbee31702a2945b1be8ef6318de80399, State: Initialized, Role: FOLLOWER
I20260923 18:14:52.028162 14646 consensus_queue.cc:260] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17acbd4b957141008f6c40974f9dfe95" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 40601 } } peers { permanent_uuid: "e003850825e44bc6a5e020118b3521c6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36253 } } peers { permanent_uuid: "cbee31702a2945b1be8ef6318de80399" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43443 } }
I20260923 18:14:52.028324 14647 consensus_queue.cc:260] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17acbd4b957141008f6c40974f9dfe95" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 40601 } } peers { permanent_uuid: "e003850825e44bc6a5e020118b3521c6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36253 } } peers { permanent_uuid: "cbee31702a2945b1be8ef6318de80399" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43443 } }
I20260923 18:14:52.028324 14648 consensus_queue.cc:260] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17acbd4b957141008f6c40974f9dfe95" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 40601 } } peers { permanent_uuid: "e003850825e44bc6a5e020118b3521c6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36253 } } peers { permanent_uuid: "cbee31702a2945b1be8ef6318de80399" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43443 } }
I20260923 18:14:52.029033 14365 heartbeater.cc:503] Master 127.27.75.254:45505 was elected leader, sending a full tablet report...
I20260923 18:14:52.029332 14497 heartbeater.cc:503] Master 127.27.75.254:45505 was elected leader, sending a full tablet report...
I20260923 18:14:52.029757 14646 ts_tablet_manager.cc:1428] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260923 18:14:52.030083 14629 heartbeater.cc:503] Master 127.27.75.254:45505 was elected leader, sending a full tablet report...
I20260923 18:14:52.030365 14648 ts_tablet_manager.cc:1428] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260923 18:14:52.031170 14647 ts_tablet_manager.cc:1428] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
W20260923 18:14:52.064323 14366 tablet.cc:2369] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:14:52.101630 14652 raft_consensus.cc:482] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:52.101823 14652 raft_consensus.cc:504] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17acbd4b957141008f6c40974f9dfe95" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 40601 } } peers { permanent_uuid: "e003850825e44bc6a5e020118b3521c6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36253 } } peers { permanent_uuid: "cbee31702a2945b1be8ef6318de80399" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43443 } }
I20260923 18:14:52.102880 14652 leader_election.cc:288] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17acbd4b957141008f6c40974f9dfe95 (127.27.75.195:40601), cbee31702a2945b1be8ef6318de80399 (127.27.75.194:43443)
I20260923 18:14:52.106933 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:52.103193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:59067 (local address 127.27.75.195:40601)
0923 18:14:52.103341 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:52.103346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:52.103519 (+ 173us) server_negotiation.cc:407] Connection header received
0923 18:14:52.103585 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:52.103588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:52.103641 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:52.103717 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:52.103975 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.104459 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.105762 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.106330 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.106452 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:52.106503 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:52.106731 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 18:14:52.106809 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 18:14:52.106921 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:52.103099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:40601 (local address 127.27.75.193:59067)
0923 18:14:52.103341 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:52.103356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:52.103469 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:52.103734 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:52.103737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:52.103756 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:52.103869 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:52.103873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.104569 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.104573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:52.105588 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:52.105601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.106466 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.106470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:52.106578 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:52.106606 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 18:14:52.106683 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 18:14:52.106752 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260923 18:14:52.107286 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:52.103592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:33479 (local address 127.27.75.194:43443)
0923 18:14:52.103716 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:52.103720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:52.103817 (+ 97us) server_negotiation.cc:407] Connection header received
0923 18:14:52.103941 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:52.103943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:52.103990 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:52.104049 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:52.104291 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.104788 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.106115 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.106654 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.106759 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:52.106800 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:52.107107 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 18:14:52.107180 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 18:14:52.107621 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:52.103469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:43443 (local address 127.27.75.193:33479)
0923 18:14:52.103745 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:52.103757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:52.103841 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:52.104066 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:52.104068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:52.104083 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:52.104197 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:52.104201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.104895 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.104898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:52.105943 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:52.105954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.106867 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.106871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:52.106970 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:52.106992 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:14:52.107410 (+ 418us) client_negotiation.cc:240] Negotiation successful
0923 18:14:52.107487 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
I20260923 18:14:52.108553 14450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20adebb204bc44d482a1f10f06e8f71f" candidate_uuid: "e003850825e44bc6a5e020118b3521c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbee31702a2945b1be8ef6318de80399" is_pre_election: true
I20260923 18:14:52.108557 14583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20adebb204bc44d482a1f10f06e8f71f" candidate_uuid: "e003850825e44bc6a5e020118b3521c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17acbd4b957141008f6c40974f9dfe95" is_pre_election: true
I20260923 18:14:52.108830 14583 raft_consensus.cc:2454] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e003850825e44bc6a5e020118b3521c6 in term 0.
I20260923 18:14:52.108837 14450 raft_consensus.cc:2454] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e003850825e44bc6a5e020118b3521c6 in term 0.
I20260923 18:14:52.109303 14254 leader_election.cc:302] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17acbd4b957141008f6c40974f9dfe95, e003850825e44bc6a5e020118b3521c6; no voters:
I20260923 18:14:52.109565 14652 raft_consensus.cc:2781] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:14:52.109649 14652 raft_consensus.cc:482] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:14:52.109696 14652 raft_consensus.cc:3037] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:52.110669 14652 raft_consensus.cc:504] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17acbd4b957141008f6c40974f9dfe95" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 40601 } } peers { permanent_uuid: "e003850825e44bc6a5e020118b3521c6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36253 } } peers { permanent_uuid: "cbee31702a2945b1be8ef6318de80399" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43443 } }
I20260923 18:14:52.110996 14652 leader_election.cc:288] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [CANDIDATE]: Term 1 election: Requested vote from peers 17acbd4b957141008f6c40974f9dfe95 (127.27.75.195:40601), cbee31702a2945b1be8ef6318de80399 (127.27.75.194:43443)
I20260923 18:14:52.111326 14583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20adebb204bc44d482a1f10f06e8f71f" candidate_uuid: "e003850825e44bc6a5e020118b3521c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17acbd4b957141008f6c40974f9dfe95"
I20260923 18:14:52.111414 14450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20adebb204bc44d482a1f10f06e8f71f" candidate_uuid: "e003850825e44bc6a5e020118b3521c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbee31702a2945b1be8ef6318de80399"
I20260923 18:14:52.111449 14583 raft_consensus.cc:3037] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:52.111548 14450 raft_consensus.cc:3037] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:52.112532 14583 raft_consensus.cc:2454] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e003850825e44bc6a5e020118b3521c6 in term 1.
I20260923 18:14:52.112620 14450 raft_consensus.cc:2454] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e003850825e44bc6a5e020118b3521c6 in term 1.
I20260923 18:14:52.112818 14254 leader_election.cc:302] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17acbd4b957141008f6c40974f9dfe95, e003850825e44bc6a5e020118b3521c6; no voters:
I20260923 18:14:52.113024 14652 raft_consensus.cc:2781] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:52.113281 14652 raft_consensus.cc:686] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 1 LEADER]: Becoming Leader. State: Replica: e003850825e44bc6a5e020118b3521c6, State: Running, Role: LEADER
I20260923 18:14:52.113626 14652 consensus_queue.cc:237] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17acbd4b957141008f6c40974f9dfe95" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 40601 } } peers { permanent_uuid: "e003850825e44bc6a5e020118b3521c6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36253 } } peers { permanent_uuid: "cbee31702a2945b1be8ef6318de80399" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43443 } }
I20260923 18:14:52.116927 14175 catalog_manager.cc:5495] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e003850825e44bc6a5e020118b3521c6 (127.27.75.193). New cstate: current_term: 1 leader_uuid: "e003850825e44bc6a5e020118b3521c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17acbd4b957141008f6c40974f9dfe95" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e003850825e44bc6a5e020118b3521c6" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbee31702a2945b1be8ef6318de80399" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43443 } health_report { overall_health: UNKNOWN } } }
I20260923 18:14:52.131197 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:52.127794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.27.75.193:36253)
0923 18:14:52.127954 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:52.127958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:52.127972 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:14:52.128039 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:52.128042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:52.128102 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:52.128193 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:14:52.128394 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.128965 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.129881 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:52.130105 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:52.130188 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:52.130423 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:14:52.130826 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:14:52.130935 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:52.131012 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 18:14:52.131046 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 18:14:52.137553 14450 raft_consensus.cc:1261] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399 [term 1 FOLLOWER]: Refusing update from remote peer e003850825e44bc6a5e020118b3521c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:52.137553 14583 raft_consensus.cc:1261] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95 [term 1 FOLLOWER]: Refusing update from remote peer e003850825e44bc6a5e020118b3521c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:52.138335 14652 consensus_queue.cc:1035] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17acbd4b957141008f6c40974f9dfe95" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:52.138588 14657 consensus_queue.cc:1035] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbee31702a2945b1be8ef6318de80399" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:52.146976 14662 mvcc.cc:205] Tried to move back new op lower bound from 7332607148586921984 to 7332607148500017152. Current Snapshot: MvccSnapshot[applied={T|T < 7332607148586921984}]
I20260923 18:14:52.147236 14661 mvcc.cc:205] Tried to move back new op lower bound from 7332607148586921984 to 7332607148500017152. Current Snapshot: MvccSnapshot[applied={T|T < 7332607148586921984}]
I20260923 18:14:52.154640 14175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:52.154794 14175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:52.157397 14175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e980392825874f76a134a435f86abe20: Sending DeleteTablet for 3 replicas of tablet 20adebb204bc44d482a1f10f06e8f71f
I20260923 18:14:52.158234 14298 tablet_service.cc:1514] Processing DeleteTablet for tablet 20adebb204bc44d482a1f10f06e8f71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:52 UTC) from {username='slave'} at 127.0.0.1:36782
I20260923 18:14:52.158314 14430 tablet_service.cc:1514] Processing DeleteTablet for tablet 20adebb204bc44d482a1f10f06e8f71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:52 UTC) from {username='slave'} at 127.0.0.1:55002
I20260923 18:14:52.158744 14670 tablet_replica.cc:330] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6: stopping tablet replica
I20260923 18:14:52.158841 14563 tablet_service.cc:1514] Processing DeleteTablet for tablet 20adebb204bc44d482a1f10f06e8f71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:52 UTC) from {username='slave'} at 127.0.0.1:40758
I20260923 18:14:52.159150 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 14236
I20260923 18:14:52.159337 14670 raft_consensus.cc:2229] T 20adebb204bc44d482a1f10f06e8f71f P e003850825e44bc6a5e020118b3521c6 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:14:52.160113 14671 tablet_replica.cc:330] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399: stopping tablet replica
I20260923 18:14:52.160436 14672 tablet_replica.cc:330] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95: stopping tablet replica
I20260923 18:14:52.160441 14671 raft_consensus.cc:2229] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:52.160777 14671 raft_consensus.cc:2258] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:52.160781 14672 raft_consensus.cc:2229] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:52.161150 14672 raft_consensus.cc:2258] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:52.162700 14672 ts_tablet_manager.cc:1905] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:52.164655 14671 ts_tablet_manager.cc:1905] T 20adebb204bc44d482a1f10f06e8f71f P cbee31702a2945b1be8ef6318de80399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:52.165284 14672 ts_tablet_manager.cc:1918] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:52.165371 14672 log.cc:1196] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/20adebb204bc44d482a1f10f06e8f71f
I20260923 18:14:52.165674 14672 ts_tablet_manager.cc:1939] T 20adebb204bc44d482a1f10f06e8f71f P 17acbd4b957141008f6c40974f9dfe95: Deleting consensus metadata
W20260923 18:14:52.166312 14161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:36253 (error 108)
I20260923 18:14:52.166610 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 14368
I20260923 18:14:52.168051 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:52.167770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36253 (local address 127.0.0.1:36796)
0923 18:14:52.167900 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:52.167974 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 18:14:52.168062 14162 catalog_manager.cc:4841] TS 17acbd4b957141008f6c40974f9dfe95 (127.27.75.195:40601): tablet 20adebb204bc44d482a1f10f06e8f71f (table test-table [id=9f1ac7d776404d42aadf605b1039ad4e]) successfully deleted
W20260923 18:14:52.168216 14161 catalog_manager.cc:4584] TS e003850825e44bc6a5e020118b3521c6 (127.27.75.193:36253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20adebb204bc44d482a1f10f06e8f71f: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36253: connect: Connection refused (error 111)
I20260923 18:14:52.172178 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 14501
I20260923 18:14:52.173308 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:52.172390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:43443 (local address 127.0.0.1:55006)
0923 18:14:52.173152 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:52.173215 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.194:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":657}
I20260923 18:14:52.177196 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 14144
2026-09-23T18:14:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1088 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:14:52 dist-test-slave-wsx5 krb5kdc[14676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:14:52 dist-test-slave-wsx5 krb5kdc[14676](info): set up 2 sockets
Sep 23 18:14:52 dist-test-slave-wsx5 krb5kdc[14676](info): commencing operation
krb5kdc: starting...
W20260923 18:14:53.163210 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:14:53 dist-test-slave-wsx5 krb5kdc[14676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:14:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:14:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:14:53.288057 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:32785
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:32785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:53.400071 14689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:53.400358 14689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:53.400416 14689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:53.404546 14689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:14:53.404620 14689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:53.404647 14689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:53.404667 14689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:14:53.404688 14689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:14:53.410225 14689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:32785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:32785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:53.411357 14689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:53.412395 14689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:53.421867 14695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:53.421989 14694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:53.422118 14697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:53.421914 14689 server_base.cc:1034] running on GCE node
I20260923 18:14:53.422806 14689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:53.423992 14689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:53.425210 14689 hybrid_clock.cc:648] HybridClock initialized: now 1790187293425191 us; error 37 us; skew 500 ppm
Sep 23 18:14:53 dist-test-slave-wsx5 krb5kdc[14676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:53.428038 14689 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:14:53.429037 14689 webserver.cc:460] Webserver started at http://127.27.75.254:45165/ using document root <none> and password file <none>
I20260923 18:14:53.429644 14689 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:53.429715 14689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:53.429939 14689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:53.432076 14689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "1163e7363bd140a49eac9388a6f6e879"
format_stamp: "Formatted at 2026-09-23 18:14:53 on dist-test-slave-wsx5"
server_key: "6629f3eb3a3e121bb023490f48fc576f"
server_key_iv: "20eb1fdc84cf1787daa3ec26bef2aae6"
server_key_version: "encryptionkey@0"
I20260923 18:14:53.432650 14689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "1163e7363bd140a49eac9388a6f6e879"
format_stamp: "Formatted at 2026-09-23 18:14:53 on dist-test-slave-wsx5"
server_key: "6629f3eb3a3e121bb023490f48fc576f"
server_key_iv: "20eb1fdc84cf1787daa3ec26bef2aae6"
server_key_version: "encryptionkey@0"
I20260923 18:14:53.436681 14689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:14:53.439568 14704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:53.441076 14689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 18:14:53.441265 14689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "1163e7363bd140a49eac9388a6f6e879"
format_stamp: "Formatted at 2026-09-23 18:14:53 on dist-test-slave-wsx5"
server_key: "6629f3eb3a3e121bb023490f48fc576f"
server_key_iv: "20eb1fdc84cf1787daa3ec26bef2aae6"
server_key_version: "encryptionkey@0"
I20260923 18:14:53.441387 14689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:53.464720 14689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:53.467885 14689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:53.468070 14689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:53.475338 14689 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:32785
I20260923 18:14:53.475353 14756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:32785 every 8 connection(s)
I20260923 18:14:53.476281 14689 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:14:53.479110 14757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:53.484680 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 14689
I20260923 18:14:53.484825 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:14:53.484869 14757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879: Bootstrap starting.
I20260923 18:14:53.485102 27951 external_mini_cluster.cc:1428] Setting key 4c03d9c1101438319a09632562d67d45
I20260923 18:14:53.487471 14757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:53.488209 14757 log.cc:824] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:53.490335 14757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879: No bootstrap required, opened a new log
Sep 23 18:14:53 dist-test-slave-wsx5 krb5kdc[14676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:53.494762 14757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1163e7363bd140a49eac9388a6f6e879" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 32785 } }
I20260923 18:14:53.495062 14757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:53.495136 14757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1163e7363bd140a49eac9388a6f6e879, State: Initialized, Role: FOLLOWER
I20260923 18:14:53.495647 14757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1163e7363bd140a49eac9388a6f6e879" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 32785 } }
I20260923 18:14:53.495795 14757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:14:53.495869 14757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:14:53.495975 14757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:53.496932 14757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1163e7363bd140a49eac9388a6f6e879" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 32785 } }
I20260923 18:14:53.497356 14757 leader_election.cc:302] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1163e7363bd140a49eac9388a6f6e879; no voters:
I20260923 18:14:53.497735 14757 leader_election.cc:288] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:14:53.497860 14762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:53.498106 14762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [term 1 LEADER]: Becoming Leader. State: Replica: 1163e7363bd140a49eac9388a6f6e879, State: Running, Role: LEADER
I20260923 18:14:53.498445 14762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1163e7363bd140a49eac9388a6f6e879" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 32785 } }
I20260923 18:14:53.498979 14757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:14:53.499770 14762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1163e7363bd140a49eac9388a6f6e879. Latest consensus state: current_term: 1 leader_uuid: "1163e7363bd140a49eac9388a6f6e879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1163e7363bd140a49eac9388a6f6e879" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 32785 } } }
I20260923 18:14:53.499897 14762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:53.500336 14764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1163e7363bd140a49eac9388a6f6e879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1163e7363bd140a49eac9388a6f6e879" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 32785 } } }
I20260923 18:14:53.500468 14764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879 [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:53.500905 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:53.486505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.27.75.254:32785)
0923 18:14:53.486921 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:53.486936 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:53.486973 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:14:53.487667 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:53.487692 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:53.488062 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:53.488386 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:53.489267 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:53.490297 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:53.491473 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:53.491964 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:53.492095 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:53.495000 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:53.495005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:53.495020 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:53.495054 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:53.496755 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:53.497305 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:53.497310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:53.497316 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:53.497437 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:53.497805 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:53.497809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:53.497811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:53.498486 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:53.499323 (+ 837us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:53.499489 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 18:14:53.499879 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260923 18:14:53.500264 14771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:14:53.504809 14771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:14:53.510349 14771 catalog_manager.cc:1347] Generated new cluster ID: ceb9f4cbbb89468fa7b4316535ee8b34
I20260923 18:14:53.510435 14771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:14:53.519665 14771 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:14:53.520642 14771 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:14:53.525043 14771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879: Generated new TSK 0
I20260923 18:14:53.525770 14771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
Entry for principal kudu/127.27.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab.
I20260923 18:14:53.564505 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:32785
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:53.680055 14783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:53.680351 14783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:53.680395 14783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:53.684554 14783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:53.684633 14783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:53.684693 14783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:14:53.690274 14783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.193.keytab
--principal=kudu/127.27.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:32785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:53.691465 14783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:53.692487 14783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:14:53.702514 14783 server_base.cc:1034] running on GCE node
W20260923 18:14:53.702539 14788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:53.702555 14791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:53.702487 14789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:53.703186 14783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:53.703801 14783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:53.704978 14783 hybrid_clock.cc:648] HybridClock initialized: now 1790187293704956 us; error 32 us; skew 500 ppm
Sep 23 18:14:53 dist-test-slave-wsx5 krb5kdc[14676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:53.707707 14783 init.cc:377] Logged in from keytab as kudu/127.27.75.193@KRBTEST.COM (short username kudu)
I20260923 18:14:53.708637 14783 webserver.cc:460] Webserver started at http://127.27.75.193:37843/ using document root <none> and password file <none>
I20260923 18:14:53.709233 14783 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:53.709307 14783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:53.709513 14783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:53.711666 14783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "d7d6c5f95deb49d29363c0aec9d47040"
format_stamp: "Formatted at 2026-09-23 18:14:53 on dist-test-slave-wsx5"
server_key: "ef3fc7503d1ce65f068b83c16e02fa41"
server_key_iv: "58220f11b2a5af414925006a749e402a"
server_key_version: "encryptionkey@0"
I20260923 18:14:53.712246 14783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "d7d6c5f95deb49d29363c0aec9d47040"
format_stamp: "Formatted at 2026-09-23 18:14:53 on dist-test-slave-wsx5"
server_key: "ef3fc7503d1ce65f068b83c16e02fa41"
server_key_iv: "58220f11b2a5af414925006a749e402a"
server_key_version: "encryptionkey@0"
I20260923 18:14:53.716133 14783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:14:53.718614 14798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:53.719882 14783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:14:53.720037 14783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "d7d6c5f95deb49d29363c0aec9d47040"
format_stamp: "Formatted at 2026-09-23 18:14:53 on dist-test-slave-wsx5"
server_key: "ef3fc7503d1ce65f068b83c16e02fa41"
server_key_iv: "58220f11b2a5af414925006a749e402a"
server_key_version: "encryptionkey@0"
I20260923 18:14:53.720140 14783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:53.734695 14783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:53.738101 14783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:53.738261 14783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:53.738818 14783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:53.739643 14783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:53.739691 14783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:53.739729 14783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:53.739753 14783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:53.748816 14783 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:36895
I20260923 18:14:53.748852 14911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:36895 every 8 connection(s)
I20260923 18:14:53.749820 14783 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:14:53.751415 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 14783
I20260923 18:14:53.751541 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:14:53.751859 27951 external_mini_cluster.cc:1428] Setting key c515ed7a1736cc752ca1a9eb4428d06b
Sep 23 18:14:53 dist-test-slave-wsx5 krb5kdc[14676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.193@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.194@KRBTEST.COM" created.
I20260923 18:14:53.764918 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:53.751830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:58325 (local address 127.27.75.254:32785)
0923 18:14:53.751962 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:53.751966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:53.753078 (+ 1112us) server_negotiation.cc:407] Connection header received
0923 18:14:53.754597 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:53.754600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:53.754656 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:53.754730 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:53.756086 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:53.756639 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:53.758292 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:53.758556 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:53.758650 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:53.761270 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:53.761273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:53.761275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:53.761303 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:53.762585 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:53.763197 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:53.763199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:53.763201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:53.763281 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:53.763743 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:53.763746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:53.763747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:53.763980 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:53.764087 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:53.764617 (+ 530us) server_negotiation.cc:299] Negotiation successful
0923 18:14:53.764752 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:14:53.765740 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:53.752313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:32785 (local address 127.27.75.193:58325)
0923 18:14:53.752915 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:53.752962 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:53.754362 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:53.754912 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:53.754920 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:53.755315 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:53.755904 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:53.755936 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:53.756768 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:53.756771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:53.758159 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:53.758171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:53.758718 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:53.758722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:53.758899 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:53.759551 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:53.759569 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:53.761062 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:53.762717 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:53.762725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:53.762743 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:53.763093 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:53.763360 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:53.763363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:53.763365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:53.763601 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:53.764092 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:53.764098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:53.764364 (+ 266us) client_negotiation.cc:769] Sending connection context
0923 18:14:53.764531 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 18:14:53.764812 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":145,"threads_started":1}
I20260923 18:14:53.767095 14912 heartbeater.cc:347] Connected to a master server at 127.27.75.254:32785
I20260923 18:14:53.767344 14912 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:53.767887 14912 heartbeater.cc:511] Master 127.27.75.254:32785 requested a full tablet report, sending...
I20260923 18:14:53.769586 14721 ts_manager.cc:194] Registered new tserver with Master: d7d6c5f95deb49d29363c0aec9d47040 (127.27.75.193:36895)
I20260923 18:14:53.770730 14721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.193@KRBTEST.COM'} at 127.27.75.193:58325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
Entry for principal kudu/127.27.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab.
I20260923 18:14:53.781517 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:32785
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:53.891759 14918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:53.892169 14918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:53.892318 14918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:53.896615 14918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:53.896684 14918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:53.896795 14918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:14:53.902443 14918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.194.keytab
--principal=kudu/127.27.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:32785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:53.903514 14918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:53.904516 14918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:53.914134 14923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:53.914170 14924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:53.914134 14926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:53.914486 14918 server_base.cc:1034] running on GCE node
I20260923 18:14:53.914986 14918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:53.915656 14918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:53.916834 14918 hybrid_clock.cc:648] HybridClock initialized: now 1790187293916811 us; error 33 us; skew 500 ppm
Sep 23 18:14:53 dist-test-slave-wsx5 krb5kdc[14676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:53.919684 14918 init.cc:377] Logged in from keytab as kudu/127.27.75.194@KRBTEST.COM (short username kudu)
I20260923 18:14:53.920632 14918 webserver.cc:460] Webserver started at http://127.27.75.194:33991/ using document root <none> and password file <none>
I20260923 18:14:53.921218 14918 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:53.921283 14918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:53.921478 14918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:53.923597 14918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "9d1117d359ed433393c0e2cdd25d3f70"
format_stamp: "Formatted at 2026-09-23 18:14:53 on dist-test-slave-wsx5"
server_key: "bc06191b888b65b1ffa3623e07574bef"
server_key_iv: "db68dc247cb1f633e56139bdc06b0de8"
server_key_version: "encryptionkey@0"
I20260923 18:14:53.924170 14918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "9d1117d359ed433393c0e2cdd25d3f70"
format_stamp: "Formatted at 2026-09-23 18:14:53 on dist-test-slave-wsx5"
server_key: "bc06191b888b65b1ffa3623e07574bef"
server_key_iv: "db68dc247cb1f633e56139bdc06b0de8"
server_key_version: "encryptionkey@0"
I20260923 18:14:53.927976 14918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 18:14:53.930511 14933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:53.931816 14918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:14:53.931939 14918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "9d1117d359ed433393c0e2cdd25d3f70"
format_stamp: "Formatted at 2026-09-23 18:14:53 on dist-test-slave-wsx5"
server_key: "bc06191b888b65b1ffa3623e07574bef"
server_key_iv: "db68dc247cb1f633e56139bdc06b0de8"
server_key_version: "encryptionkey@0"
I20260923 18:14:53.932047 14918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:53.946470 14918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:53.949805 14918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:53.949987 14918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:53.950584 14918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:53.951406 14918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:53.951462 14918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:53.951516 14918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:53.951539 14918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:53.960107 14918 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:42187
I20260923 18:14:53.960131 15047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:42187 every 8 connection(s)
I20260923 18:14:53.961014 14918 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:14:53.967926 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 14918
I20260923 18:14:53.968063 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:14:53.968335 27951 external_mini_cluster.cc:1428] Setting key 962c3331a2a14f9bd58948142d7d61c5
Sep 23 18:14:53 dist-test-slave-wsx5 krb5kdc[14676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.194@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:53.975199 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:53.962873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:51493 (local address 127.27.75.254:32785)
0923 18:14:53.962998 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:53.963001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:53.963885 (+ 884us) server_negotiation.cc:407] Connection header received
0923 18:14:53.965236 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:53.965239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:53.965291 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:53.965367 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:53.966538 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:53.967097 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:53.968284 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:53.968550 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:53.968638 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:53.971209 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:53.971213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:53.971216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:53.971244 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:53.972715 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:53.973339 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:53.973343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:53.973344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:53.973431 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:53.973787 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:53.973790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:53.973791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:53.974023 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:53.974123 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:53.974936 (+ 813us) server_negotiation.cc:299] Negotiation successful
0923 18:14:53.975072 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 18:14:53.976353 15050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:53.963283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:32785 (local address 127.27.75.194:51493)
0923 18:14:53.963718 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:53.963764 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:53.965008 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:53.965539 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:53.965547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:53.965898 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:53.966371 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:53.966384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:53.967213 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:53.967216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:53.968165 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:53.968176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:53.968707 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:53.968711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:53.968875 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:53.969531 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:53.969547 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:53.971041 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:53.972859 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:53.972864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:53.972880 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:53.973223 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:53.973529 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:53.973531 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:53.973533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:53.973684 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:53.974144 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:53.974152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:53.974531 (+ 379us) client_negotiation.cc:769] Sending connection context
0923 18:14:53.974785 (+ 254us) client_negotiation.cc:240] Negotiation successful
0923 18:14:53.975228 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":108,"threads_started":1}
I20260923 18:14:53.977646 15048 heartbeater.cc:347] Connected to a master server at 127.27.75.254:32785
I20260923 18:14:53.977952 15048 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.75.195@KRBTEST.COM; defaulting to no policy
I20260923 18:14:53.978585 15048 heartbeater.cc:511] Master 127.27.75.254:32785 requested a full tablet report, sending...
I20260923 18:14:53.979677 14721 ts_manager.cc:194] Registered new tserver with Master: 9d1117d359ed433393c0e2cdd25d3f70 (127.27.75.194:42187)
I20260923 18:14:53.980346 14721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.194@KRBTEST.COM'} at 127.27.75.194:51493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
Entry for principal kudu/127.27.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab.
I20260923 18:14:53.996048 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:32785
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:54.107864 15053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:54.108160 15053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:54.108223 15053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:54.112424 15053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:54.112502 15053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:54.112589 15053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:14:54.119064 15053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.195.keytab
--principal=kudu/127.27.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:32785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:54.120170 15053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:54.121255 15053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:54.131013 15058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:54.131059 15053 server_base.cc:1034] running on GCE node
W20260923 18:14:54.131011 15061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:54.131011 15059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:54.131764 15053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:54.132419 15053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:54.133596 15053 hybrid_clock.cc:648] HybridClock initialized: now 1790187294133563 us; error 49 us; skew 500 ppm
Sep 23 18:14:54 dist-test-slave-wsx5 krb5kdc[14676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:54.136498 15053 init.cc:377] Logged in from keytab as kudu/127.27.75.195@KRBTEST.COM (short username kudu)
I20260923 18:14:54.137482 15053 webserver.cc:460] Webserver started at http://127.27.75.195:36913/ using document root <none> and password file <none>
I20260923 18:14:54.138162 15053 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:54.138266 15053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:54.138486 15053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:54.140648 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "71d2251b54b149d38462aba32daec309"
format_stamp: "Formatted at 2026-09-23 18:14:54 on dist-test-slave-wsx5"
server_key: "ed9a48c2c0f930e08f3bafae77c02271"
server_key_iv: "f835e95daddcc811e29d05cd2ffaa605"
server_key_version: "encryptionkey@0"
I20260923 18:14:54.141278 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "71d2251b54b149d38462aba32daec309"
format_stamp: "Formatted at 2026-09-23 18:14:54 on dist-test-slave-wsx5"
server_key: "ed9a48c2c0f930e08f3bafae77c02271"
server_key_iv: "f835e95daddcc811e29d05cd2ffaa605"
server_key_version: "encryptionkey@0"
I20260923 18:14:54.145284 15053 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:14:54.148065 15069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:54.149514 15053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:54.149639 15053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "71d2251b54b149d38462aba32daec309"
format_stamp: "Formatted at 2026-09-23 18:14:54 on dist-test-slave-wsx5"
server_key: "ed9a48c2c0f930e08f3bafae77c02271"
server_key_iv: "f835e95daddcc811e29d05cd2ffaa605"
server_key_version: "encryptionkey@0"
I20260923 18:14:54.149751 15053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:54.172636 15053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:54.175990 15053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:54.176178 15053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:54.176781 15053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:54.177666 15053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:54.177714 15053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:54.177769 15053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:54.177798 15053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:54.186702 15053 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:35629
I20260923 18:14:54.186726 15182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:35629 every 8 connection(s)
I20260923 18:14:54.187620 15053 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:14:54.192859 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 15053
I20260923 18:14:54.193035 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:14:54.193372 27951 external_mini_cluster.cc:1428] Setting key c7b062e8ead31acaa51185845dea085b
Sep 23 18:14:54 dist-test-slave-wsx5 krb5kdc[14676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.195@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:14:54.201349 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.189471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:36101 (local address 127.27.75.254:32785)
0923 18:14:54.189601 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:54.189605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:54.190432 (+ 827us) server_negotiation.cc:407] Connection header received
0923 18:14:54.191729 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:54.191732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:54.191787 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:54.191868 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:54.193039 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.193780 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.195105 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.195369 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.195483 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.197905 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:54.197908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:54.197910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:54.197938 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:54.199345 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:54.199788 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:54.199790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:54.199791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:54.199863 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:54.200194 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:54.200196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:54.200197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:54.200428 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:54.200509 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:54.201082 (+ 573us) server_negotiation.cc:299] Negotiation successful
0923 18:14:54.201213 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:14:54.202255 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.189863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:32785 (local address 127.27.75.195:36101)
0923 18:14:54.190284 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:54.190325 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:54.191511 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:54.192020 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:54.192027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:54.192365 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:54.192878 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.192892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.193947 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.193949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.194988 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.194998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.195531 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.195535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.195714 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.196241 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:14:54.196256 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:54.197747 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:54.199448 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:54.199453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:54.199465 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:54.199713 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:54.199979 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:14:54.199982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:14:54.199984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:14:54.200121 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:14:54.200554 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:54.200560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:54.200847 (+ 287us) client_negotiation.cc:769] Sending connection context
0923 18:14:54.201022 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 18:14:54.201377 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
I20260923 18:14:54.203437 15183 heartbeater.cc:347] Connected to a master server at 127.27.75.254:32785
I20260923 18:14:54.203677 15183 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:54.204155 15183 heartbeater.cc:511] Master 127.27.75.254:32785 requested a full tablet report, sending...
I20260923 18:14:54.205307 14721 ts_manager.cc:194] Registered new tserver with Master: 71d2251b54b149d38462aba32daec309 (127.27.75.195:35629)
I20260923 18:14:54.205884 14721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.75.195@KRBTEST.COM'} at 127.27.75.195:36101
I20260923 18:14:54.207181 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:14:54.216475 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.209245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.27.75.254:32785)
0923 18:14:54.209418 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:54.209423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:54.209472 (+ 49us) server_negotiation.cc:407] Connection header received
0923 18:14:54.209594 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:54.209596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:54.209634 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:54.209696 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:54.210394 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.210912 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.211693 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.211892 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.211960 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.213245 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:54.213248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:54.213251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:54.213276 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:54.214587 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:54.215051 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:54.215054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:54.215055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:54.215160 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:54.215491 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:54.215495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:54.215496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:54.215755 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:54.215877 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:54.216187 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 18:14:54.216345 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 18:14:54.219846 14720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:14:54.222996 14720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:14:54.238468 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.232436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:42187 (local address 127.0.0.1:53008)
0923 18:14:54.233653 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:54.233665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:54.233743 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:54.234196 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:54.234199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:54.234214 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:54.234326 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.234332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.235364 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.235369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.236798 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.236809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.237884 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.237888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.238020 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.238062 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 18:14:54.238170 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 18:14:54.238302 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"spinlock_wait_cycles":8960,"thread_start_us":91,"threads_started":1}
I20260923 18:14:54.238623 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.232936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.27.75.194:42187)
0923 18:14:54.233350 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:54.233361 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:54.233753 (+ 392us) server_negotiation.cc:407] Connection header received
0923 18:14:54.233827 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:54.233831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:54.233978 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:54.234096 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:54.234440 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.235246 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.236918 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.237746 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.237866 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.238028 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:54.238348 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 18:14:54.238471 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":104,"threads_started":1}
I20260923 18:14:54.238776 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.232936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.27.75.193:36895)
0923 18:14:54.233528 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:54.233542 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:54.233565 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:14:54.233642 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:54.233648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:54.233873 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:54.234025 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:54.234318 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.235578 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.237216 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.237804 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.237905 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.238050 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:54.238543 (+ 493us) server_negotiation.cc:299] Negotiation successful
0923 18:14:54.238646 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":63,"threads_started":1}
I20260923 18:14:54.239051 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.232238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36895 (local address 127.0.0.1:49862)
0923 18:14:54.233358 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:54.233372 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:54.233477 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:54.234022 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:54.234029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:54.234050 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:54.234191 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.234198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.235731 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.235734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.237050 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.237061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.238110 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.238113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.238276 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.238299 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 18:14:54.238358 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 18:14:54.238430 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":79,"thread_start_us":122,"threads_started":1}
I20260923 18:14:54.239050 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.232085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:35629 (local address 127.0.0.1:44858)
0923 18:14:54.232964 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:54.233021 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:54.233168 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:54.234195 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:54.234198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:54.234232 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:54.234348 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.234355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.235731 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.235734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.237051 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.237061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.238123 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.238127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.238301 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.238325 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:14:54.238394 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 18:14:54.238479 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":137,"spinlock_wait_cycles":1792,"thread_start_us":80,"threads_started":1}
I20260923 18:14:54.239794 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.232228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.27.75.195:35629)
0923 18:14:54.233602 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:54.233610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:54.233629 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:14:54.233689 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:54.233692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:54.233878 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:54.234036 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:54.234478 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.235591 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.237195 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.238017 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.238120 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.238256 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:54.239531 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0923 18:14:54.239658 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1282,"thread_start_us":143,"threads_started":1}
I20260923 18:14:54.241828 14981 tablet_service.cc:1467] Processing CreateTablet for tablet e70bc1e20a1b42669e4237b668019538 (DEFAULT_TABLE table=test-table [id=4c423c87f8e34a8d9b0a17a21cf2b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:54.242209 15117 tablet_service.cc:1467] Processing CreateTablet for tablet e70bc1e20a1b42669e4237b668019538 (DEFAULT_TABLE table=test-table [id=4c423c87f8e34a8d9b0a17a21cf2b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:54.242275 14846 tablet_service.cc:1467] Processing CreateTablet for tablet e70bc1e20a1b42669e4237b668019538 (DEFAULT_TABLE table=test-table [id=4c423c87f8e34a8d9b0a17a21cf2b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:54.243076 14981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70bc1e20a1b42669e4237b668019538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:54.243176 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70bc1e20a1b42669e4237b668019538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:54.243216 14846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e70bc1e20a1b42669e4237b668019538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:54.249974 15201 tablet_bootstrap.cc:492] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: Bootstrap starting.
I20260923 18:14:54.251986 15203 tablet_bootstrap.cc:492] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: Bootstrap starting.
I20260923 18:14:54.252620 15201 tablet_bootstrap.cc:654] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:54.253527 15202 tablet_bootstrap.cc:492] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040: Bootstrap starting.
I20260923 18:14:54.254078 15201 log.cc:824] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:54.254495 15203 tablet_bootstrap.cc:654] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:54.255754 15202 tablet_bootstrap.cc:654] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:54.255790 15203 log.cc:824] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:54.257200 15202 log.cc:824] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:54.257392 15201 tablet_bootstrap.cc:492] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: No bootstrap required, opened a new log
I20260923 18:14:54.257612 15201 ts_tablet_manager.cc:1397] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 18:14:54.257642 15203 tablet_bootstrap.cc:492] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: No bootstrap required, opened a new log
I20260923 18:14:54.257822 15203 ts_tablet_manager.cc:1397] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 18:14:54.259338 15202 tablet_bootstrap.cc:492] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040: No bootstrap required, opened a new log
I20260923 18:14:54.259541 15202 ts_tablet_manager.cc:1397] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:14:54.261008 15203 raft_consensus.cc:348] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d2251b54b149d38462aba32daec309" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35629 } } peers { permanent_uuid: "d7d6c5f95deb49d29363c0aec9d47040" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36895 } } peers { permanent_uuid: "9d1117d359ed433393c0e2cdd25d3f70" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 42187 } }
I20260923 18:14:54.261323 15203 raft_consensus.cc:374] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:54.261384 15203 raft_consensus.cc:729] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1117d359ed433393c0e2cdd25d3f70, State: Initialized, Role: FOLLOWER
I20260923 18:14:54.261610 15201 raft_consensus.cc:348] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d2251b54b149d38462aba32daec309" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35629 } } peers { permanent_uuid: "d7d6c5f95deb49d29363c0aec9d47040" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36895 } } peers { permanent_uuid: "9d1117d359ed433393c0e2cdd25d3f70" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 42187 } }
I20260923 18:14:54.261883 15201 raft_consensus.cc:374] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:54.261931 15201 raft_consensus.cc:729] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d2251b54b149d38462aba32daec309, State: Initialized, Role: FOLLOWER
I20260923 18:14:54.261905 15203 consensus_queue.cc:260] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d2251b54b149d38462aba32daec309" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35629 } } peers { permanent_uuid: "d7d6c5f95deb49d29363c0aec9d47040" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36895 } } peers { permanent_uuid: "9d1117d359ed433393c0e2cdd25d3f70" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 42187 } }
I20260923 18:14:54.262355 15201 consensus_queue.cc:260] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d2251b54b149d38462aba32daec309" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35629 } } peers { permanent_uuid: "d7d6c5f95deb49d29363c0aec9d47040" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36895 } } peers { permanent_uuid: "9d1117d359ed433393c0e2cdd25d3f70" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 42187 } }
I20260923 18:14:54.263095 15203 ts_tablet_manager.cc:1428] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 18:14:54.263345 15201 ts_tablet_manager.cc:1428] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 18:14:54.263427 15048 heartbeater.cc:503] Master 127.27.75.254:32785 was elected leader, sending a full tablet report...
I20260923 18:14:54.263428 15183 heartbeater.cc:503] Master 127.27.75.254:32785 was elected leader, sending a full tablet report...
I20260923 18:14:54.263615 15202 raft_consensus.cc:348] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d2251b54b149d38462aba32daec309" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35629 } } peers { permanent_uuid: "d7d6c5f95deb49d29363c0aec9d47040" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36895 } } peers { permanent_uuid: "9d1117d359ed433393c0e2cdd25d3f70" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 42187 } }
I20260923 18:14:54.263860 15202 raft_consensus.cc:374] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:54.263896 15202 raft_consensus.cc:729] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d6c5f95deb49d29363c0aec9d47040, State: Initialized, Role: FOLLOWER
I20260923 18:14:54.264253 15202 consensus_queue.cc:260] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d2251b54b149d38462aba32daec309" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35629 } } peers { permanent_uuid: "d7d6c5f95deb49d29363c0aec9d47040" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36895 } } peers { permanent_uuid: "9d1117d359ed433393c0e2cdd25d3f70" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 42187 } }
I20260923 18:14:54.265111 14912 heartbeater.cc:503] Master 127.27.75.254:32785 was elected leader, sending a full tablet report...
I20260923 18:14:54.265254 15202 ts_tablet_manager.cc:1428] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260923 18:14:54.436376 15207 raft_consensus.cc:482] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:54.436625 15207 raft_consensus.cc:504] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d2251b54b149d38462aba32daec309" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35629 } } peers { permanent_uuid: "d7d6c5f95deb49d29363c0aec9d47040" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36895 } } peers { permanent_uuid: "9d1117d359ed433393c0e2cdd25d3f70" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 42187 } }
W20260923 18:14:54.438839 15184 tablet.cc:2369] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:14:54.439462 15207 leader_election.cc:288] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d2251b54b149d38462aba32daec309 (127.27.75.195:35629), d7d6c5f95deb49d29363c0aec9d47040 (127.27.75.193:36895)
I20260923 18:14:54.442962 15050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.438236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:35629 (local address 127.27.75.194:47131)
0923 18:14:54.438362 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:54.438378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:54.438478 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:54.439007 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:54.439011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:54.439032 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:54.439145 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.439151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.440282 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.440285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.441573 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.441581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.442514 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.442518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.442633 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.442660 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 18:14:54.442735 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 18:14:54.442814 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 18:14:54.442983 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.438550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:47131 (local address 127.27.75.195:35629)
0923 18:14:54.438666 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:54.438670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:54.438684 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:14:54.438739 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:54.438742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:54.438792 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:54.438865 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:54.439266 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.440144 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.441714 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.442372 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.442521 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.442586 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:54.442765 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 18:14:54.442849 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:14:54.443725 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.438715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:43825 (local address 127.27.75.193:36895)
0923 18:14:54.438827 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:54.438831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:54.439863 (+ 1032us) server_negotiation.cc:407] Connection header received
0923 18:14:54.440062 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:54.440065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:54.440120 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:54.440165 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:54.440623 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.441417 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.442524 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.443094 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.443189 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.443237 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:54.443550 (+ 313us) server_negotiation.cc:299] Negotiation successful
0923 18:14:54.443622 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:14:54.443720 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.438290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36895 (local address 127.27.75.194:43825)
0923 18:14:54.439753 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:54.439770 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:54.439942 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:54.440308 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:54.440311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:54.440329 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:54.440479 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.440486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.441581 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.441584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.442381 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:54.442391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.443327 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.443331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:54.443423 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.443445 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:14:54.443515 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 18:14:54.443586 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"thread_start_us":110,"threads_started":1}
I20260923 18:14:54.443832 15137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70bc1e20a1b42669e4237b668019538" candidate_uuid: "9d1117d359ed433393c0e2cdd25d3f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d2251b54b149d38462aba32daec309" is_pre_election: true
I20260923 18:14:54.444180 15137 raft_consensus.cc:2454] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1117d359ed433393c0e2cdd25d3f70 in term 0.
I20260923 18:14:54.444325 14866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70bc1e20a1b42669e4237b668019538" candidate_uuid: "9d1117d359ed433393c0e2cdd25d3f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d6c5f95deb49d29363c0aec9d47040" is_pre_election: true
I20260923 18:14:54.444610 14866 raft_consensus.cc:2454] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d1117d359ed433393c0e2cdd25d3f70 in term 0.
I20260923 18:14:54.444711 14935 leader_election.cc:302] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d2251b54b149d38462aba32daec309, 9d1117d359ed433393c0e2cdd25d3f70; no voters:
I20260923 18:14:54.445081 15207 raft_consensus.cc:2781] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:14:54.445190 15207 raft_consensus.cc:482] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:14:54.445246 15207 raft_consensus.cc:3037] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:54.446398 15207 raft_consensus.cc:504] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d2251b54b149d38462aba32daec309" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35629 } } peers { permanent_uuid: "d7d6c5f95deb49d29363c0aec9d47040" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36895 } } peers { permanent_uuid: "9d1117d359ed433393c0e2cdd25d3f70" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 42187 } }
I20260923 18:14:54.446753 15207 leader_election.cc:288] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [CANDIDATE]: Term 1 election: Requested vote from peers 71d2251b54b149d38462aba32daec309 (127.27.75.195:35629), d7d6c5f95deb49d29363c0aec9d47040 (127.27.75.193:36895)
I20260923 18:14:54.447103 15137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70bc1e20a1b42669e4237b668019538" candidate_uuid: "9d1117d359ed433393c0e2cdd25d3f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d2251b54b149d38462aba32daec309"
I20260923 18:14:54.447206 14866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e70bc1e20a1b42669e4237b668019538" candidate_uuid: "9d1117d359ed433393c0e2cdd25d3f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d6c5f95deb49d29363c0aec9d47040"
I20260923 18:14:54.447247 15137 raft_consensus.cc:3037] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:54.447320 14866 raft_consensus.cc:3037] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:54.448553 15137 raft_consensus.cc:2454] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1117d359ed433393c0e2cdd25d3f70 in term 1.
I20260923 18:14:54.448553 14866 raft_consensus.cc:2454] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d1117d359ed433393c0e2cdd25d3f70 in term 1.
I20260923 18:14:54.448911 14934 leader_election.cc:302] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d1117d359ed433393c0e2cdd25d3f70, d7d6c5f95deb49d29363c0aec9d47040; no voters:
I20260923 18:14:54.449110 15207 raft_consensus.cc:2781] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:54.449340 15207 raft_consensus.cc:686] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 1 LEADER]: Becoming Leader. State: Replica: 9d1117d359ed433393c0e2cdd25d3f70, State: Running, Role: LEADER
I20260923 18:14:54.449707 15207 consensus_queue.cc:237] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d2251b54b149d38462aba32daec309" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35629 } } peers { permanent_uuid: "d7d6c5f95deb49d29363c0aec9d47040" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36895 } } peers { permanent_uuid: "9d1117d359ed433393c0e2cdd25d3f70" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 42187 } }
I20260923 18:14:54.452877 14721 catalog_manager.cc:5495] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d1117d359ed433393c0e2cdd25d3f70 (127.27.75.194). New cstate: current_term: 1 leader_uuid: "9d1117d359ed433393c0e2cdd25d3f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d2251b54b149d38462aba32daec309" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d6c5f95deb49d29363c0aec9d47040" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d1117d359ed433393c0e2cdd25d3f70" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 42187 } health_report { overall_health: HEALTHY } } }
W20260923 18:14:54.462119 15049 tablet.cc:2369] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:14:54.475906 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.472245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.27.75.194:42187)
0923 18:14:54.472378 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:54.472383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:54.472397 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:14:54.472498 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:54.472501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:54.472556 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:54.472650 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:14:54.472903 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.473464 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.474334 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.474555 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.474641 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.474940 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:14:54.475466 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:14:54.475582 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:54.475695 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 18:14:54.475760 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 18:14:54.483110 15137 raft_consensus.cc:1261] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309 [term 1 FOLLOWER]: Refusing update from remote peer 9d1117d359ed433393c0e2cdd25d3f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:54.483110 14866 raft_consensus.cc:1261] T e70bc1e20a1b42669e4237b668019538 P d7d6c5f95deb49d29363c0aec9d47040 [term 1 FOLLOWER]: Refusing update from remote peer 9d1117d359ed433393c0e2cdd25d3f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:54.483873 15211 consensus_queue.cc:1035] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d6c5f95deb49d29363c0aec9d47040" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:54.484066 15207 consensus_queue.cc:1035] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d2251b54b149d38462aba32daec309" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:54.495744 15216 mvcc.cc:205] Tried to move back new op lower bound from 7332607158194868224 to 7332607158069084160. Current Snapshot: MvccSnapshot[applied={T|T < 7332607158194868224}]
I20260923 18:14:54.495955 15215 mvcc.cc:205] Tried to move back new op lower bound from 7332607158194868224 to 7332607158069084160. Current Snapshot: MvccSnapshot[applied={T|T < 7332607158194868224}]
I20260923 18:14:54.503342 14721 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:54.503535 14721 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:54.506183 14721 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1163e7363bd140a49eac9388a6f6e879: Sending DeleteTablet for 3 replicas of tablet e70bc1e20a1b42669e4237b668019538
I20260923 18:14:54.507062 14846 tablet_service.cc:1514] Processing DeleteTablet for tablet e70bc1e20a1b42669e4237b668019538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:54 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:49862
I20260923 18:14:54.507161 15117 tablet_service.cc:1514] Processing DeleteTablet for tablet e70bc1e20a1b42669e4237b668019538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:54 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:44858
I20260923 18:14:54.507156 14981 tablet_service.cc:1514] Processing DeleteTablet for tablet e70bc1e20a1b42669e4237b668019538 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:54 UTC) from {username='kudu', principal='kudu/127.27.75.254@KRBTEST.COM'} at 127.0.0.1:53008
I20260923 18:14:54.507659 15225 tablet_replica.cc:330] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: stopping tablet replica
I20260923 18:14:54.507926 15224 tablet_replica.cc:330] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: stopping tablet replica
I20260923 18:14:54.508013 15225 raft_consensus.cc:2229] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:14:54.508227 15224 raft_consensus.cc:2229] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:54.508443 15225 raft_consensus.cc:2258] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:54.508528 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 14783
I20260923 18:14:54.508564 15224 raft_consensus.cc:2258] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:54.510404 15224 ts_tablet_manager.cc:1905] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:54.510403 15225 ts_tablet_manager.cc:1905] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:54.513206 15225 ts_tablet_manager.cc:1918] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:54.513206 15224 ts_tablet_manager.cc:1918] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:54.513310 15225 log.cc:1196] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/e70bc1e20a1b42669e4237b668019538
I20260923 18:14:54.513314 15224 log.cc:1196] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/e70bc1e20a1b42669e4237b668019538
I20260923 18:14:54.513598 15225 ts_tablet_manager.cc:1939] T e70bc1e20a1b42669e4237b668019538 P 9d1117d359ed433393c0e2cdd25d3f70: Deleting consensus metadata
I20260923 18:14:54.513621 15224 ts_tablet_manager.cc:1939] T e70bc1e20a1b42669e4237b668019538 P 71d2251b54b149d38462aba32daec309: Deleting consensus metadata
I20260923 18:14:54.514771 14707 catalog_manager.cc:4841] TS 9d1117d359ed433393c0e2cdd25d3f70 (127.27.75.194:42187): tablet e70bc1e20a1b42669e4237b668019538 (table test-table [id=4c423c87f8e34a8d9b0a17a21cf2b9f6]) successfully deleted
I20260923 18:14:54.514771 14706 catalog_manager.cc:4841] TS 71d2251b54b149d38462aba32daec309 (127.27.75.195:35629): tablet e70bc1e20a1b42669e4237b668019538 (table test-table [id=4c423c87f8e34a8d9b0a17a21cf2b9f6]) successfully deleted
I20260923 18:14:54.518102 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 14918
W20260923 18:14:54.518147 14705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:36895 (error 108)
I20260923 18:14:54.519426 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.519137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:36895 (local address 127.0.0.1:49878)
0923 18:14:54.519269 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:54.519334 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 18:14:54.519603 14705 catalog_manager.cc:4584] TS d7d6c5f95deb49d29363c0aec9d47040 (127.27.75.193:36895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e70bc1e20a1b42669e4237b668019538: Network error: Client connection negotiation failed: client connection to 127.27.75.193:36895: connect: Connection refused (error 111)
I20260923 18:14:54.523588 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 15053
I20260923 18:14:54.528486 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 14689
2026-09-23T18:14:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2352 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T18:14:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:14:54Z Disabled control of system clock
I20260923 18:14:54.557801 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42929
--webserver_interface=127.27.75.254
--webserver_port=0
--builtin_ntp_servers=127.27.75.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:42929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:14:54.669628 15230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:54.669927 15230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:54.669963 15230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:54.674067 15230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:14:54.674132 15230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:54.674146 15230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:54.674162 15230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:14:54.674176 15230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:14:54.679131 15230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:42929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:42929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:54.680249 15230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:54.681743 15230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:54.690442 15235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:54.690452 15238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:54.690501 15236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:54.690704 15230 server_base.cc:1034] running on GCE node
I20260923 18:14:54.691457 15230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:54.692636 15230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:54.693823 15230 hybrid_clock.cc:648] HybridClock initialized: now 1790187294693811 us; error 28 us; skew 500 ppm
I20260923 18:14:54.696007 15230 webserver.cc:460] Webserver started at http://127.27.75.254:37603/ using document root <none> and password file <none>
I20260923 18:14:54.696595 15230 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:54.696663 15230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:54.696857 15230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:54.699045 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "e8d4097ecb4b4ddabfaaff6c93ecb91c"
format_stamp: "Formatted at 2026-09-23 18:14:54 on dist-test-slave-wsx5"
server_key: "751eec5d80c46db6f769af3974e0fb06"
server_key_iv: "834996ca69630dd7e1eb786bb2d371ee"
server_key_version: "encryptionkey@0"
I20260923 18:14:54.699618 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "e8d4097ecb4b4ddabfaaff6c93ecb91c"
format_stamp: "Formatted at 2026-09-23 18:14:54 on dist-test-slave-wsx5"
server_key: "751eec5d80c46db6f769af3974e0fb06"
server_key_iv: "834996ca69630dd7e1eb786bb2d371ee"
server_key_version: "encryptionkey@0"
I20260923 18:14:54.703584 15230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:14:54.706211 15244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:54.707527 15230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:54.707653 15230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "e8d4097ecb4b4ddabfaaff6c93ecb91c"
format_stamp: "Formatted at 2026-09-23 18:14:54 on dist-test-slave-wsx5"
server_key: "751eec5d80c46db6f769af3974e0fb06"
server_key_iv: "834996ca69630dd7e1eb786bb2d371ee"
server_key_version: "encryptionkey@0"
I20260923 18:14:54.707760 15230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:54.722168 15230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:54.722900 15230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:54.723202 15230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:54.732715 15230 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:42929
I20260923 18:14:54.732766 15296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:42929 every 8 connection(s)
I20260923 18:14:54.733961 15230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:14:54.737269 15297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:54.743528 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 15230
I20260923 18:14:54.743673 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:14:54.743990 27951 external_mini_cluster.cc:1428] Setting key 5f34c677aaee479cdd4385135ecad12c
I20260923 18:14:54.744347 15297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c: Bootstrap starting.
I20260923 18:14:54.747557 15297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:54.748517 15297 log.cc:824] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:54.751020 15297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c: No bootstrap required, opened a new log
I20260923 18:14:54.755843 15297 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d4097ecb4b4ddabfaaff6c93ecb91c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42929 } }
I20260923 18:14:54.756129 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:54.745696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.27.75.254:42929)
0923 18:14:54.746476 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:54.746494 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:54.746538 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:14:54.747411 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:54.747437 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:54.748003 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:54.748412 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:54.748885 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.750432 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.751704 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:54.752388 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:54.753010 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:54.753598 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:54.753604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:54.753622 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:54.754147 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:54.754222 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:54.754262 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:54.754720 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:54.754806 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 18:14:54.755127 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":140,"threads_started":1}
I20260923 18:14:54.756132 15297 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:54.756209 15297 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d4097ecb4b4ddabfaaff6c93ecb91c, State: Initialized, Role: FOLLOWER
I20260923 18:14:54.756664 15297 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d4097ecb4b4ddabfaaff6c93ecb91c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42929 } }
I20260923 18:14:54.756876 15297 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:14:54.756948 15297 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:14:54.757042 15297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:54.758180 15297 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d4097ecb4b4ddabfaaff6c93ecb91c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42929 } }
I20260923 18:14:54.758538 15297 leader_election.cc:302] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8d4097ecb4b4ddabfaaff6c93ecb91c; no voters:
I20260923 18:14:54.758891 15297 leader_election.cc:288] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:14:54.758983 15302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:54.759193 15302 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [term 1 LEADER]: Becoming Leader. State: Replica: e8d4097ecb4b4ddabfaaff6c93ecb91c, State: Running, Role: LEADER
I20260923 18:14:54.759486 15302 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d4097ecb4b4ddabfaaff6c93ecb91c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42929 } }
I20260923 18:14:54.760056 15297 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:14:54.761457 15304 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8d4097ecb4b4ddabfaaff6c93ecb91c. Latest consensus state: current_term: 1 leader_uuid: "e8d4097ecb4b4ddabfaaff6c93ecb91c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d4097ecb4b4ddabfaaff6c93ecb91c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42929 } } }
I20260923 18:14:54.761582 15304 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:54.761869 15303 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8d4097ecb4b4ddabfaaff6c93ecb91c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d4097ecb4b4ddabfaaff6c93ecb91c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 42929 } } }
I20260923 18:14:54.761960 15303 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:54.762020 15311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:14:54.766055 15311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:14:54.772171 15311 catalog_manager.cc:1347] Generated new cluster ID: 1bfd6388d26d440181a2a4141607b1c0
I20260923 18:14:54.772255 15311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:14:54.797798 15311 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:14:54.798791 15311 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:14:54.807818 15311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c: Generated new TSK 0
I20260923 18:14:54.808740 15311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:14:54.818699 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.193:0
--local_ip_for_outbound_sockets=127.27.75.193
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42929
--builtin_ntp_servers=127.27.75.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 18:14:54.933830 15321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:54.934118 15321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:54.934175 15321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:54.938354 15321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:54.938427 15321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:54.938508 15321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.193
I20260923 18:14:54.943480 15321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.75.193
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.193
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:54.944653 15321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:54.946111 15321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:54.956036 15327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:54.955979 15326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:54.956148 15321 server_base.cc:1034] running on GCE node
W20260923 18:14:54.955981 15329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:54.956841 15321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:54.957535 15321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:54.958719 15321 hybrid_clock.cc:648] HybridClock initialized: now 1790187294958699 us; error 29 us; skew 500 ppm
I20260923 18:14:54.965293 15321 webserver.cc:460] Webserver started at http://127.27.75.193:34849/ using document root <none> and password file <none>
I20260923 18:14:54.965932 15321 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:54.966020 15321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:54.966207 15321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:54.968449 15321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/data/instance:
uuid: "841406accdee496face540d477dd858d"
format_stamp: "Formatted at 2026-09-23 18:14:54 on dist-test-slave-wsx5"
server_key: "fe739a745ff9e39deb64a109afd08fd1"
server_key_iv: "c93e45e30aec0dd597b12d14fffd310d"
server_key_version: "encryptionkey@0"
I20260923 18:14:54.969043 15321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance:
uuid: "841406accdee496face540d477dd858d"
format_stamp: "Formatted at 2026-09-23 18:14:54 on dist-test-slave-wsx5"
server_key: "fe739a745ff9e39deb64a109afd08fd1"
server_key_iv: "c93e45e30aec0dd597b12d14fffd310d"
server_key_version: "encryptionkey@0"
I20260923 18:14:54.973237 15321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:14:54.976037 15335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:54.977615 15321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:54.977730 15321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/wal
uuid: "841406accdee496face540d477dd858d"
format_stamp: "Formatted at 2026-09-23 18:14:54 on dist-test-slave-wsx5"
server_key: "fe739a745ff9e39deb64a109afd08fd1"
server_key_iv: "c93e45e30aec0dd597b12d14fffd310d"
server_key_version: "encryptionkey@0"
I20260923 18:14:54.977850 15321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:54.987730 15321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:54.988397 15321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:54.988548 15321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:54.989110 15321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:54.990099 15321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:54.990145 15321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:54.990185 15321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:54.990206 15321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:54.998616 15321 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.193:34687
I20260923 18:14:54.998658 15449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.193:34687 every 8 connection(s)
I20260923 18:14:54.999562 15321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/data/info.pb
I20260923 18:14:55.005343 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 15321
I20260923 18:14:55.005461 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-0/wal/instance
I20260923 18:14:55.005709 27951 external_mini_cluster.cc:1428] Setting key d459b05e75d3c9b7c14e8b2385faa5fb
I20260923 18:14:55.007558 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.194:0
--local_ip_for_outbound_sockets=127.27.75.194
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42929
--builtin_ntp_servers=127.27.75.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:14:55.009189 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.001454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.193:34495 (local address 127.27.75.254:42929)
0923 18:14:55.001614 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:55.001619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:55.002724 (+ 1105us) server_negotiation.cc:407] Connection header received
0923 18:14:55.004330 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:55.004333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:55.004383 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:55.004470 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:55.005460 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.006039 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.007313 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.007532 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.007619 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.008683 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:55.008687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:55.008690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:55.008704 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:55.008724 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:55.008732 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:55.008790 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:55.008944 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 18:14:55.009013 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 18:14:55.010123 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.001877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42929 (local address 127.27.75.193:34495)
0923 18:14:55.002575 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:55.002616 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:55.004109 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:55.004660 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:55.004667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:55.004755 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:55.005290 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.005332 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.006169 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.006171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.007202 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.007213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.007672 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.007676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.007837 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.008416 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:14:55.008417 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:55.008460 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:14:55.008532 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:55.008809 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:55.008819 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:55.008825 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:14:55.008935 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 18:14:55.009359 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
I20260923 18:14:55.011343 15450 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42929
I20260923 18:14:55.011574 15450 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:55.012179 15450 heartbeater.cc:511] Master 127.27.75.254:42929 requested a full tablet report, sending...
I20260923 18:14:55.013751 15261 ts_manager.cc:194] Registered new tserver with Master: 841406accdee496face540d477dd858d (127.27.75.193:34687)
I20260923 18:14:55.014990 15261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.193:34495
W20260923 18:14:55.120781 15453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:55.121079 15453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:55.121157 15453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:55.125349 15453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:55.125418 15453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:55.125499 15453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.194
I20260923 18:14:55.130395 15453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.75.194
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.194
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:55.131486 15453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:55.132891 15453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:55.142769 15459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:55.142774 15461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:55.142997 15453 server_base.cc:1034] running on GCE node
W20260923 18:14:55.142756 15458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:55.143622 15453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:55.144240 15453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:55.145395 15453 hybrid_clock.cc:648] HybridClock initialized: now 1790187295145381 us; error 26 us; skew 500 ppm
I20260923 18:14:55.147310 15453 webserver.cc:460] Webserver started at http://127.27.75.194:43409/ using document root <none> and password file <none>
I20260923 18:14:55.147933 15453 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:55.148000 15453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:55.148200 15453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:55.150312 15453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/data/instance:
uuid: "af572dd8a348426d9a4013d56fda60ad"
format_stamp: "Formatted at 2026-09-23 18:14:55 on dist-test-slave-wsx5"
server_key: "2d3b3f5876ac868ec35a3247eab54b27"
server_key_iv: "77c251f3143db5573cda07f4a2a8def0"
server_key_version: "encryptionkey@0"
I20260923 18:14:55.150882 15453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance:
uuid: "af572dd8a348426d9a4013d56fda60ad"
format_stamp: "Formatted at 2026-09-23 18:14:55 on dist-test-slave-wsx5"
server_key: "2d3b3f5876ac868ec35a3247eab54b27"
server_key_iv: "77c251f3143db5573cda07f4a2a8def0"
server_key_version: "encryptionkey@0"
I20260923 18:14:55.154642 15453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:14:55.157341 15467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:55.158655 15453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 18:14:55.158792 15453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/wal
uuid: "af572dd8a348426d9a4013d56fda60ad"
format_stamp: "Formatted at 2026-09-23 18:14:55 on dist-test-slave-wsx5"
server_key: "2d3b3f5876ac868ec35a3247eab54b27"
server_key_iv: "77c251f3143db5573cda07f4a2a8def0"
server_key_version: "encryptionkey@0"
I20260923 18:14:55.158911 15453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:55.180004 15453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:55.180784 15453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:55.180989 15453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:55.181644 15453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:55.182636 15453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:55.182682 15453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:55.182750 15453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:55.182777 15453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:55.191078 15453 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.194:36969
I20260923 18:14:55.191144 15581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.194:36969 every 8 connection(s)
I20260923 18:14:55.191998 15453 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/data/info.pb
I20260923 18:14:55.193996 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 15453
I20260923 18:14:55.194129 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/wal/instance
I20260923 18:14:55.194437 27951 external_mini_cluster.cc:1428] Setting key 071115725c86aca4e970186dc09f610d
I20260923 18:14:55.196333 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.75.195:0
--local_ip_for_outbound_sockets=127.27.75.195
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42929
--builtin_ntp_servers=127.27.75.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 18:14:55.201725 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.194030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.194:52525 (local address 127.27.75.254:42929)
0923 18:14:55.194208 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:55.194214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:55.195337 (+ 1123us) server_negotiation.cc:407] Connection header received
0923 18:14:55.196783 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:55.196787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:55.196842 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:55.196929 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:55.198004 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.198536 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.199710 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.199937 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.200036 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.201162 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:55.201166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:55.201169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:55.201185 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:55.201207 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:55.201216 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:55.201278 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:55.201497 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 18:14:55.201570 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 18:14:55.202626 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.194453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42929 (local address 127.27.75.194:52525)
0923 18:14:55.195178 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:55.195234 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:55.196560 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:55.197182 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:55.197190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:55.197285 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:55.197811 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.197825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.198675 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.198678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.199584 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.199594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.200101 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.200105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.200263 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.200858 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:14:55.200860 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:55.200903 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:14:55.200977 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:55.201289 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:55.201299 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:55.201305 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 18:14:55.201479 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 18:14:55.201793 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260923 18:14:55.203842 15582 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42929
I20260923 18:14:55.204092 15582 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:55.204622 15582 heartbeater.cc:511] Master 127.27.75.254:42929 requested a full tablet report, sending...
I20260923 18:14:55.205782 15261 ts_manager.cc:194] Registered new tserver with Master: af572dd8a348426d9a4013d56fda60ad (127.27.75.194:36969)
I20260923 18:14:55.206549 15261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.194:52525
W20260923 18:14:55.309183 15585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:55.309470 15585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:55.309508 15585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:55.313783 15585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:55.313854 15585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:55.313911 15585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.75.195
I20260923 18:14:55.319126 15585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.75.195
--webserver_port=0
--tserver_master_addrs=127.27.75.254:42929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.75.195
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:55.320303 15585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:55.321797 15585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:14:55.331377 15585 server_base.cc:1034] running on GCE node
W20260923 18:14:55.331367 15593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:55.331422 15590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:55.331374 15591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:55.332154 15585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:55.332872 15585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:55.334067 15585 hybrid_clock.cc:648] HybridClock initialized: now 1790187295334030 us; error 49 us; skew 500 ppm
I20260923 18:14:55.335974 15585 webserver.cc:460] Webserver started at http://127.27.75.195:41613/ using document root <none> and password file <none>
I20260923 18:14:55.336536 15585 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:55.336604 15585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:55.336798 15585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:55.338847 15585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/data/instance:
uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967"
format_stamp: "Formatted at 2026-09-23 18:14:55 on dist-test-slave-wsx5"
server_key: "9e76a52a672dd56d8c8c01494f38f7b4"
server_key_iv: "c8cf83e8e31fff90f814b6e8c6b7af43"
server_key_version: "encryptionkey@0"
I20260923 18:14:55.339372 15585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance:
uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967"
format_stamp: "Formatted at 2026-09-23 18:14:55 on dist-test-slave-wsx5"
server_key: "9e76a52a672dd56d8c8c01494f38f7b4"
server_key_iv: "c8cf83e8e31fff90f814b6e8c6b7af43"
server_key_version: "encryptionkey@0"
I20260923 18:14:55.343027 15585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 18:14:55.345535 15599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:55.346769 15585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:14:55.346886 15585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/wal
uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967"
format_stamp: "Formatted at 2026-09-23 18:14:55 on dist-test-slave-wsx5"
server_key: "9e76a52a672dd56d8c8c01494f38f7b4"
server_key_iv: "c8cf83e8e31fff90f814b6e8c6b7af43"
server_key_version: "encryptionkey@0"
I20260923 18:14:55.347008 15585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:55.371230 15585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:55.371989 15585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:55.372164 15585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:55.372779 15585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 18:14:55.373711 15585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 18:14:55.373759 15585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:55.373811 15585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 18:14:55.373840 15585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:55.382201 15585 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.195:36363
I20260923 18:14:55.382231 15713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.195:36363 every 8 connection(s)
I20260923 18:14:55.383157 15585 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/data/info.pb
I20260923 18:14:55.392246 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.385084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:41471 (local address 127.27.75.254:42929)
0923 18:14:55.385277 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:55.385282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:55.386327 (+ 1045us) server_negotiation.cc:407] Connection header received
0923 18:14:55.387674 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:55.387676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:55.387724 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:55.387796 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:55.388686 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.389242 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.390401 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.390629 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.390696 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.391863 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:55.391866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:55.391869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:55.391881 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:55.391900 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:55.391908 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:55.391963 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:55.392077 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 18:14:55.392129 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 18:14:55.392760 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 15585
I20260923 18:14:55.392846 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/wal/instance
I20260923 18:14:55.393158 27951 external_mini_cluster.cc:1428] Setting key b45c8f004d07ff47a6a62b636512dd9e
I20260923 18:14:55.393244 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.385496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:42929 (local address 127.27.75.195:41471)
0923 18:14:55.386169 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:55.386216 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:55.387488 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:55.387951 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:55.387957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:55.388041 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:14:55.388541 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.388554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.389374 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.389377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.390306 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.390315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.390808 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.390812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.390987 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.391609 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 18:14:55.391610 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:14:55.391651 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 18:14:55.391728 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:14:55.391971 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:14:55.391976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:14:55.391980 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 18:14:55.392073 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 18:14:55.392305 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":98,"threads_started":1}
I20260923 18:14:55.394330 15714 heartbeater.cc:347] Connected to a master server at 127.27.75.254:42929
I20260923 18:14:55.394565 15714 heartbeater.cc:464] Registering TS with master...
I20260923 18:14:55.395087 15714 heartbeater.cc:511] Master 127.27.75.254:42929 requested a full tablet report, sending...
I20260923 18:14:55.396076 15260 ts_manager.cc:194] Registered new tserver with Master: 2ddd8cc9b84e4c6e9d9b4b30f7215967 (127.27.75.195:36363)
I20260923 18:14:55.396705 15260 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.75.195:41471
I20260923 18:14:55.407235 27951 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 18:14:55.412766 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.409273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.27.75.254:42929)
0923 18:14:55.409399 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:55.409419 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:55.409575 (+ 156us) server_negotiation.cc:407] Connection header received
0923 18:14:55.409678 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:55.409680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:55.409722 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:55.409789 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:55.410047 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.410563 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.411375 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.411609 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.411680 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.412356 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:55.412360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:55.412362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:55.412373 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:55.412390 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:55.412397 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:55.412469 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:55.412584 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 18:14:55.412636 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 18:14:55.415565 15260 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 18:14:55.418651 15260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 18:14:55.433314 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.427170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.195:36363 (local address 127.0.0.1:50960)
0923 18:14:55.427534 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:55.427587 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:55.427718 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:55.428636 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:55.428641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:55.428665 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:55.428803 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.428810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.430255 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.430259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.431641 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.431651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.432715 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.432718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.432804 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.432837 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 18:14:55.432977 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 18:14:55.433144 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":45,"thread_start_us":97,"threads_started":1}
I20260923 18:14:55.433557 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.428305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.27.75.194:36969)
0923 18:14:55.428721 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:55.428735 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:55.428757 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:14:55.428831 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:55.428835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:55.429008 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:55.429185 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:55.429374 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.430399 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.432053 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.432848 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.432955 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.433149 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:55.433288 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 18:14:55.433390 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":114,"threads_started":1}
I20260923 18:14:55.433866 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.427491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:36969 (local address 127.0.0.1:57720)
0923 18:14:55.427723 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:55.427734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:55.427807 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:55.429158 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:55.429161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:55.429178 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:55.429271 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.429277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.430505 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.430509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.431913 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.431925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.433029 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.433033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.433142 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.433164 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 18:14:55.433240 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 18:14:55.433309 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":86,"thread_start_us":95,"threads_started":1}
I20260923 18:14:55.433990 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.427269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.27.75.195:36363)
0923 18:14:55.428114 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:55.428127 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:55.428150 (+ 23us) server_negotiation.cc:407] Connection header received
0923 18:14:55.428231 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:55.428237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:55.428467 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:55.428629 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:55.428927 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.430129 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.431762 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.432562 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.432674 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.432908 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:55.433731 (+ 823us) server_negotiation.cc:299] Negotiation successful
0923 18:14:55.433836 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":165,"threads_started":1}
I20260923 18:14:55.434304 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.427891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:34687 (local address 127.0.0.1:60486)
0923 18:14:55.428103 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:55.428114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:55.428186 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:55.429291 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:55.429294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:55.429309 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:55.429394 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.429399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.430505 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.430509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.431940 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.431946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.433936 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.433939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.434033 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.434053 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 18:14:55.434110 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 18:14:55.434176 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":96,"thread_start_us":86,"threads_started":1}
I20260923 18:14:55.436627 15647 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5de763d7ff4a6fbcfc06342851d0f7 (DEFAULT_TABLE table=test-table [id=bdef6f23b29b41218377849783610d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:55.436622 15515 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5de763d7ff4a6fbcfc06342851d0f7 (DEFAULT_TABLE table=test-table [id=bdef6f23b29b41218377849783610d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:55.437570 15647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5de763d7ff4a6fbcfc06342851d0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:55.437572 15515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5de763d7ff4a6fbcfc06342851d0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:55.438455 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.428360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.27.75.193:34687)
0923 18:14:55.428895 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:55.428906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:55.428928 (+ 22us) server_negotiation.cc:407] Connection header received
0923 18:14:55.428991 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:55.428994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:55.429157 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:55.429514 (+ 357us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:55.429621 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.430376 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.433000 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.433822 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.433921 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.434699 (+ 778us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:55.434749 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 18:14:55.438277 (+ 3528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":180,"threads_started":1}
I20260923 18:14:55.441533 15384 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5de763d7ff4a6fbcfc06342851d0f7 (DEFAULT_TABLE table=test-table [id=bdef6f23b29b41218377849783610d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 18:14:55.442737 15384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5de763d7ff4a6fbcfc06342851d0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:55.444581 15732 tablet_bootstrap.cc:492] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967: Bootstrap starting.
I20260923 18:14:55.445683 15731 tablet_bootstrap.cc:492] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad: Bootstrap starting.
I20260923 18:14:55.446745 15732 tablet_bootstrap.cc:654] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:55.447476 15732 log.cc:824] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:55.447860 15731 tablet_bootstrap.cc:654] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:55.448685 15731 log.cc:824] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:55.449419 15732 tablet_bootstrap.cc:492] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967: No bootstrap required, opened a new log
I20260923 18:14:55.449632 15732 ts_tablet_manager.cc:1397] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 18:14:55.450796 15731 tablet_bootstrap.cc:492] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad: No bootstrap required, opened a new log
I20260923 18:14:55.450996 15731 ts_tablet_manager.cc:1397] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 18:14:55.451274 15735 tablet_bootstrap.cc:492] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d: Bootstrap starting.
I20260923 18:14:55.453262 15732 raft_consensus.cc:348] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 36363 } } peers { permanent_uuid: "841406accdee496face540d477dd858d" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 34687 } } peers { permanent_uuid: "af572dd8a348426d9a4013d56fda60ad" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 36969 } }
I20260923 18:14:55.453504 15732 raft_consensus.cc:374] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:55.453557 15732 raft_consensus.cc:729] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ddd8cc9b84e4c6e9d9b4b30f7215967, State: Initialized, Role: FOLLOWER
I20260923 18:14:55.453585 15735 tablet_bootstrap.cc:654] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:55.453974 15732 consensus_queue.cc:260] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 36363 } } peers { permanent_uuid: "841406accdee496face540d477dd858d" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 34687 } } peers { permanent_uuid: "af572dd8a348426d9a4013d56fda60ad" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 36969 } }
I20260923 18:14:55.454419 15735 log.cc:824] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:55.454607 15714 heartbeater.cc:503] Master 127.27.75.254:42929 was elected leader, sending a full tablet report...
I20260923 18:14:55.455039 15732 ts_tablet_manager.cc:1428] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 18:14:55.455096 15731 raft_consensus.cc:348] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 36363 } } peers { permanent_uuid: "841406accdee496face540d477dd858d" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 34687 } } peers { permanent_uuid: "af572dd8a348426d9a4013d56fda60ad" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 36969 } }
I20260923 18:14:55.455353 15731 raft_consensus.cc:374] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:55.455415 15731 raft_consensus.cc:729] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af572dd8a348426d9a4013d56fda60ad, State: Initialized, Role: FOLLOWER
I20260923 18:14:55.455834 15731 consensus_queue.cc:260] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 36363 } } peers { permanent_uuid: "841406accdee496face540d477dd858d" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 34687 } } peers { permanent_uuid: "af572dd8a348426d9a4013d56fda60ad" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 36969 } }
I20260923 18:14:55.456547 15735 tablet_bootstrap.cc:492] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d: No bootstrap required, opened a new log
I20260923 18:14:55.456734 15735 ts_tablet_manager.cc:1397] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 18:14:55.456765 15731 ts_tablet_manager.cc:1428] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 18:14:55.457095 15582 heartbeater.cc:503] Master 127.27.75.254:42929 was elected leader, sending a full tablet report...
I20260923 18:14:55.460489 15735 raft_consensus.cc:348] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 36363 } } peers { permanent_uuid: "841406accdee496face540d477dd858d" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 34687 } } peers { permanent_uuid: "af572dd8a348426d9a4013d56fda60ad" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 36969 } }
I20260923 18:14:55.460728 15735 raft_consensus.cc:374] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:55.460781 15735 raft_consensus.cc:729] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841406accdee496face540d477dd858d, State: Initialized, Role: FOLLOWER
I20260923 18:14:55.461205 15735 consensus_queue.cc:260] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 36363 } } peers { permanent_uuid: "841406accdee496face540d477dd858d" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 34687 } } peers { permanent_uuid: "af572dd8a348426d9a4013d56fda60ad" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 36969 } }
I20260923 18:14:55.461896 15450 heartbeater.cc:503] Master 127.27.75.254:42929 was elected leader, sending a full tablet report...
I20260923 18:14:55.462100 15735 ts_tablet_manager.cc:1428] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
W20260923 18:14:55.500772 15451 tablet.cc:2369] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:14:55.505825 15736 raft_consensus.cc:482] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 18:14:55.506011 15736 raft_consensus.cc:504] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 36363 } } peers { permanent_uuid: "841406accdee496face540d477dd858d" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 34687 } } peers { permanent_uuid: "af572dd8a348426d9a4013d56fda60ad" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 36969 } }
I20260923 18:14:55.507123 15736 leader_election.cc:288] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 841406accdee496face540d477dd858d (127.27.75.193:34687), af572dd8a348426d9a4013d56fda60ad (127.27.75.194:36969)
I20260923 18:14:55.510881 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.507332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:34687 (local address 127.27.75.195:54937)
0923 18:14:55.507461 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:55.507477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:55.507592 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:55.507844 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:55.507847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:55.507868 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:55.507989 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.507997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.508840 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.508843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.509759 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.509769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.510454 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.510458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.510573 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.510621 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 18:14:55.510679 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 18:14:55.510740 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":22}
I20260923 18:14:55.510922 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.507455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:54937 (local address 127.27.75.193:34687)
0923 18:14:55.507589 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:55.507593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:55.507608 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:14:55.507689 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:55.507692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:55.507741 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:55.507817 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:55.508121 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.508737 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.509890 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.510320 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.510414 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.510457 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:55.510716 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 18:14:55.510800 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 18:14:55.510998 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.507438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.75.195:36673 (local address 127.27.75.194:36969)
0923 18:14:55.507557 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:55.507562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:55.507946 (+ 384us) server_negotiation.cc:407] Connection header received
0923 18:14:55.508024 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:55.508027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:55.508077 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:55.508144 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 18:14:55.508389 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.508995 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.510027 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.510414 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.510522 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.510568 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:55.510814 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 18:14:55.510886 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 18:14:55.511039 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.507332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:36969 (local address 127.27.75.195:36673)
0923 18:14:55.507845 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:55.507858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:14:55.507953 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:14:55.508164 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:14:55.508166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:14:55.508182 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 18:14:55.508291 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.508296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.509098 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.509100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.509910 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:14:55.509920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.510529 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.510533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:14:55.510685 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.510709 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 18:14:55.510783 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 18:14:55.510903 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":185,"thread_start_us":122,"threads_started":1}
I20260923 18:14:55.511631 15536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5de763d7ff4a6fbcfc06342851d0f7" candidate_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af572dd8a348426d9a4013d56fda60ad" is_pre_election: true
I20260923 18:14:55.511618 15404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5de763d7ff4a6fbcfc06342851d0f7" candidate_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841406accdee496face540d477dd858d" is_pre_election: true
I20260923 18:14:55.511933 15404 raft_consensus.cc:2454] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ddd8cc9b84e4c6e9d9b4b30f7215967 in term 0.
I20260923 18:14:55.511927 15536 raft_consensus.cc:2454] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ddd8cc9b84e4c6e9d9b4b30f7215967 in term 0.
I20260923 18:14:55.512427 15600 leader_election.cc:302] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ddd8cc9b84e4c6e9d9b4b30f7215967, 841406accdee496face540d477dd858d; no voters:
I20260923 18:14:55.512722 15736 raft_consensus.cc:2781] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 18:14:55.512799 15736 raft_consensus.cc:482] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 18:14:55.512830 15736 raft_consensus.cc:3037] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:55.513872 15736 raft_consensus.cc:504] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 36363 } } peers { permanent_uuid: "841406accdee496face540d477dd858d" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 34687 } } peers { permanent_uuid: "af572dd8a348426d9a4013d56fda60ad" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 36969 } }
I20260923 18:14:55.514206 15736 leader_election.cc:288] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [CANDIDATE]: Term 1 election: Requested vote from peers 841406accdee496face540d477dd858d (127.27.75.193:34687), af572dd8a348426d9a4013d56fda60ad (127.27.75.194:36969)
I20260923 18:14:55.514540 15404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5de763d7ff4a6fbcfc06342851d0f7" candidate_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841406accdee496face540d477dd858d"
I20260923 18:14:55.514539 15536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5de763d7ff4a6fbcfc06342851d0f7" candidate_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af572dd8a348426d9a4013d56fda60ad"
I20260923 18:14:55.514645 15404 raft_consensus.cc:3037] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:55.514679 15536 raft_consensus.cc:3037] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:55.515582 15404 raft_consensus.cc:2454] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ddd8cc9b84e4c6e9d9b4b30f7215967 in term 1.
I20260923 18:14:55.515729 15536 raft_consensus.cc:2454] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ddd8cc9b84e4c6e9d9b4b30f7215967 in term 1.
I20260923 18:14:55.515857 15600 leader_election.cc:302] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ddd8cc9b84e4c6e9d9b4b30f7215967, 841406accdee496face540d477dd858d; no voters:
I20260923 18:14:55.516037 15736 raft_consensus.cc:2781] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:55.516258 15736 raft_consensus.cc:686] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 1 LEADER]: Becoming Leader. State: Replica: 2ddd8cc9b84e4c6e9d9b4b30f7215967, State: Running, Role: LEADER
I20260923 18:14:55.516624 15736 consensus_queue.cc:237] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 36363 } } peers { permanent_uuid: "841406accdee496face540d477dd858d" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 34687 } } peers { permanent_uuid: "af572dd8a348426d9a4013d56fda60ad" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 36969 } }
I20260923 18:14:55.519766 15261 catalog_manager.cc:5495] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ddd8cc9b84e4c6e9d9b4b30f7215967 (127.27.75.195). New cstate: current_term: 1 leader_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ddd8cc9b84e4c6e9d9b4b30f7215967" member_type: VOTER last_known_addr { host: "127.27.75.195" port: 36363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "841406accdee496face540d477dd858d" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 34687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af572dd8a348426d9a4013d56fda60ad" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 36969 } health_report { overall_health: UNKNOWN } } }
I20260923 18:14:55.533156 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.529599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.27.75.195:36363)
0923 18:14:55.529717 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:55.529720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:55.529733 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:14:55.529873 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:55.529875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:55.529928 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:55.530013 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:14:55.530258 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.530754 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.531614 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:55.531842 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:55.531931 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:55.532282 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:14:55.532738 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:14:55.532837 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:55.532911 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 18:14:55.532965 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:14:55.539269 15404 raft_consensus.cc:1261] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d [term 1 FOLLOWER]: Refusing update from remote peer 2ddd8cc9b84e4c6e9d9b4b30f7215967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:55.539259 15536 raft_consensus.cc:1261] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad [term 1 FOLLOWER]: Refusing update from remote peer 2ddd8cc9b84e4c6e9d9b4b30f7215967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:14:55.539983 15736 consensus_queue.cc:1035] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "841406accdee496face540d477dd858d" member_type: VOTER last_known_addr { host: "127.27.75.193" port: 34687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:55.540177 15741 consensus_queue.cc:1035] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af572dd8a348426d9a4013d56fda60ad" member_type: VOTER last_known_addr { host: "127.27.75.194" port: 36969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:14:55.548132 15745 mvcc.cc:205] Tried to move back new op lower bound from 7332607162521116672 to 7332607162439065600. Current Snapshot: MvccSnapshot[applied={T|T < 7332607162521116672}]
I20260923 18:14:55.556082 15261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:55.556249 15261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 18:14:55.558727 15261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8d4097ecb4b4ddabfaaff6c93ecb91c: Sending DeleteTablet for 3 replicas of tablet 4d5de763d7ff4a6fbcfc06342851d0f7
I20260923 18:14:55.559636 15515 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d5de763d7ff4a6fbcfc06342851d0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:55 UTC) from {username='slave'} at 127.0.0.1:57720
I20260923 18:14:55.559643 15384 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d5de763d7ff4a6fbcfc06342851d0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:55 UTC) from {username='slave'} at 127.0.0.1:60486
I20260923 18:14:55.560240 15754 tablet_replica.cc:330] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d: stopping tablet replica
I20260923 18:14:55.560396 15647 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d5de763d7ff4a6fbcfc06342851d0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 18:14:55 UTC) from {username='slave'} at 127.0.0.1:50960
I20260923 18:14:55.560771 15754 raft_consensus.cc:2229] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:55.560988 15756 tablet_replica.cc:330] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967: stopping tablet replica
I20260923 18:14:55.561151 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 15321
I20260923 18:14:55.561349 15754 raft_consensus.cc:2258] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 841406accdee496face540d477dd858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:55.562358 15755 tablet_replica.cc:330] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad: stopping tablet replica
I20260923 18:14:55.562788 15755 raft_consensus.cc:2229] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 18:14:55.563226 15755 raft_consensus.cc:2258] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:55.563584 15756 raft_consensus.cc:2229] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 1 LEADER]: Raft consensus shutting down.
I20260923 18:14:55.564024 15756 raft_consensus.cc:2258] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 18:14:55.565142 15755 ts_tablet_manager.cc:1905] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:55.565476 15756 ts_tablet_manager.cc:1905] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 18:14:55.568079 15755 ts_tablet_manager.cc:1918] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:55.568280 15755 log.cc:1196] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-1/wal/wals/4d5de763d7ff4a6fbcfc06342851d0f7
I20260923 18:14:55.568292 15756 ts_tablet_manager.cc:1918] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 18:14:55.568388 15756 log.cc:1196] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967: Deleting WAL directory at /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790187042487232-27951-0/minicluster-data/ts-2/wal/wals/4d5de763d7ff4a6fbcfc06342851d0f7
I20260923 18:14:55.568701 15756 ts_tablet_manager.cc:1939] T 4d5de763d7ff4a6fbcfc06342851d0f7 P 2ddd8cc9b84e4c6e9d9b4b30f7215967: Deleting consensus metadata
I20260923 18:14:55.568728 15755 ts_tablet_manager.cc:1939] T 4d5de763d7ff4a6fbcfc06342851d0f7 P af572dd8a348426d9a4013d56fda60ad: Deleting consensus metadata
W20260923 18:14:55.568877 15245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.75.193:34687 (error 108)
I20260923 18:14:55.569753 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 15453
I20260923 18:14:55.570582 15245 catalog_manager.cc:4841] TS 2ddd8cc9b84e4c6e9d9b4b30f7215967 (127.27.75.195:36363): tablet 4d5de763d7ff4a6fbcfc06342851d0f7 (table test-table [id=bdef6f23b29b41218377849783610d22]) successfully deleted
I20260923 18:14:55.574180 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.573853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.193:34687 (local address 127.0.0.1:60502)
0923 18:14:55.574005 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:55.574082 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.193:34687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 18:14:55.574363 15245 catalog_manager.cc:4584] TS 841406accdee496face540d477dd858d (127.27.75.193:34687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d5de763d7ff4a6fbcfc06342851d0f7: Network error: Client connection negotiation failed: client connection to 127.27.75.193:34687: connect: Connection refused (error 111)
I20260923 18:14:55.575915 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 15585
I20260923 18:14:55.576996 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:55.576077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.194:36969 (local address 127.0.0.1:57724)
0923 18:14:55.576849 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0923 18:14:55.576910 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.194:36969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683}
I20260923 18:14:55.581349 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 15230
2026-09-23T18:14:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1051 ms)
[----------] 6 tests from EncryptionPolicyTest (10222 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:14:55 dist-test-slave-wsx5 krb5kdc[15760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:14:55 dist-test-slave-wsx5 krb5kdc[15760](info): set up 2 sockets
Sep 23 18:14:55 dist-test-slave-wsx5 krb5kdc[15760](info): commencing operation
krb5kdc: starting...
W20260923 18:14:56.555315 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:14:56 dist-test-slave-wsx5 krb5kdc[15760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:14:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:14:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:14:56.680318 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:56.796849 15773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:56.797168 15773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:56.797232 15773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:56.801416 15773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:14:56.801496 15773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:56.801514 15773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:56.801532 15773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:14:56.801548 15773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:14:56.807165 15773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43891
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:56.808245 15773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:56.809319 15773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:56.817845 15779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:56.817901 15780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:56.817866 15782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:56.818042 15773 server_base.cc:1034] running on GCE node
I20260923 18:14:56.818940 15773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:56.820205 15773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:56.821378 15773 hybrid_clock.cc:648] HybridClock initialized: now 1790187296821361 us; error 35 us; skew 500 ppm
Sep 23 18:14:56 dist-test-slave-wsx5 krb5kdc[15760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:56.824345 15773 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:14:56.825425 15773 webserver.cc:460] Webserver started at http://127.0.0.1:41011/ using document root <none> and password file <none>
I20260923 18:14:56.825996 15773 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:56.826061 15773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:56.826260 15773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:56.828364 15773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "333169ab1abf4f8dad3c5323eb0e8c8b"
format_stamp: "Formatted at 2026-09-23 18:14:56 on dist-test-slave-wsx5"
server_key: "c09f6632221b3b12dbacbc077039601e"
server_key_iv: "af09c12998fd04e502e9b9a07c38d203"
server_key_version: "encryptionkey@0"
I20260923 18:14:56.828964 15773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "333169ab1abf4f8dad3c5323eb0e8c8b"
format_stamp: "Formatted at 2026-09-23 18:14:56 on dist-test-slave-wsx5"
server_key: "c09f6632221b3b12dbacbc077039601e"
server_key_iv: "af09c12998fd04e502e9b9a07c38d203"
server_key_version: "encryptionkey@0"
I20260923 18:14:56.832777 15773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 18:14:56.835493 15789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:56.836814 15773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:14:56.836933 15773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "333169ab1abf4f8dad3c5323eb0e8c8b"
format_stamp: "Formatted at 2026-09-23 18:14:56 on dist-test-slave-wsx5"
server_key: "c09f6632221b3b12dbacbc077039601e"
server_key_iv: "af09c12998fd04e502e9b9a07c38d203"
server_key_version: "encryptionkey@0"
I20260923 18:14:56.837056 15773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:56.850864 15773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:56.854357 15773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:56.854581 15773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:56.861590 15773 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43891
I20260923 18:14:56.861619 15841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43891 every 8 connection(s)
I20260923 18:14:56.862579 15773 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:14:56.865645 15842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:56.867030 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 15773
I20260923 18:14:56.867175 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:14:56.867477 27951 external_mini_cluster.cc:1428] Setting key eab54c1808311138f186962d5a134a34
I20260923 18:14:56.872036 15842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b: Bootstrap starting.
I20260923 18:14:56.874276 15842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:56.874979 15842 log.cc:824] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:14:56 dist-test-slave-wsx5 krb5kdc[15760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 18:14:56.877266 15842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b: No bootstrap required, opened a new log
I20260923 18:14:56.881031 15842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333169ab1abf4f8dad3c5323eb0e8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } }
I20260923 18:14:56.881332 15842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:56.881387 15842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333169ab1abf4f8dad3c5323eb0e8c8b, State: Initialized, Role: FOLLOWER
I20260923 18:14:56.881808 15842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333169ab1abf4f8dad3c5323eb0e8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } }
I20260923 18:14:56.881919 15842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:14:56.881978 15842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:14:56.882026 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:56.869098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57375 (local address 127.0.0.1:43891)
0923 18:14:56.869589 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:56.869604 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:56.869641 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:14:56.870276 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:56.870293 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:56.870648 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:56.870962 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:56.871759 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:56.872776 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:56.873672 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:56.874135 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:56.874277 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:56.877038 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:56.877044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:56.877057 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:56.877085 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:56.878852 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:56.879367 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:56.879373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:56.879378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:56.879502 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:56.879839 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:56.879843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:56.879844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:56.880304 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:56.880543 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:56.880837 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 18:14:56.881068 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":126,"threads_started":1}
I20260923 18:14:56.882069 15842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:56.882980 15842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333169ab1abf4f8dad3c5323eb0e8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } }
I20260923 18:14:56.883306 15842 leader_election.cc:302] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 333169ab1abf4f8dad3c5323eb0e8c8b; no voters:
I20260923 18:14:56.883603 15842 leader_election.cc:288] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:14:56.883703 15847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:56.883868 15847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [term 1 LEADER]: Becoming Leader. State: Replica: 333169ab1abf4f8dad3c5323eb0e8c8b, State: Running, Role: LEADER
I20260923 18:14:56.884172 15847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333169ab1abf4f8dad3c5323eb0e8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } }
I20260923 18:14:56.884903 15842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:14:56.885242 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:14:56.885820 15849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 333169ab1abf4f8dad3c5323eb0e8c8b. Latest consensus state: current_term: 1 leader_uuid: "333169ab1abf4f8dad3c5323eb0e8c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333169ab1abf4f8dad3c5323eb0e8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } } }
I20260923 18:14:56.885944 15849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:56.885926 15848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "333169ab1abf4f8dad3c5323eb0e8c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333169ab1abf4f8dad3c5323eb0e8c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } } }
I20260923 18:14:56.886152 15848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:56.886838 15856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:14:56.890024 15856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:14:56.895502 15856 catalog_manager.cc:1347] Generated new cluster ID: 6db2ed33a51d4ed29b6627a4bb0bbb60
I20260923 18:14:56.895570 15856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:14:56.897571 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:56.887448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38751 (local address 127.0.0.1:43891)
0923 18:14:56.887591 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:56.887595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:56.888157 (+ 562us) server_negotiation.cc:407] Connection header received
0923 18:14:56.888356 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:56.888360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:56.888410 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:56.888504 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:56.889358 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:56.890114 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:56.891357 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:56.891719 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:56.891798 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:56.893837 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:56.893841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:56.893844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:14:56.893879 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:56.895581 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:56.896200 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:56.896203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:56.896205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:56.896301 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:14:56.896674 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:14:56.896677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:14:56.896678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:14:56.896882 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:56.897017 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:56.897322 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 18:14:56.897421 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 18:14:56.904506 15856 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:14:56.905694 15856 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:14:56.929530 15856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 333169ab1abf4f8dad3c5323eb0e8c8b: Generated new TSK 0
I20260923 18:14:56.930541 15856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:14:56.936165 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 15773
W20260923 18:14:56.941094 15758 connection.cc:535] client connection to 127.0.0.1:43891 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43891): Connection reset by peer (error 104)
2026-09-23T18:14:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:14:56 dist-test-slave-wsx5 krb5kdc[15873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:14:56 dist-test-slave-wsx5 krb5kdc[15873](info): set up 2 sockets
Sep 23 18:14:56 dist-test-slave-wsx5 krb5kdc[15873](info): commencing operation
krb5kdc: starting...
W20260923 18:14:57.899263 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:14:57 dist-test-slave-wsx5 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:14:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:14:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:14:58.028075 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:58.140321 15886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:58.140630 15886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:58.140692 15886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:58.144874 15886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:14:58.144951 15886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:58.144979 15886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:58.144999 15886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:14:58.145030 15886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:14:58.150738 15886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:58.152089 15886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:58.153324 15886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:58.162117 15891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:58.162122 15894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:58.162159 15886 server_base.cc:1034] running on GCE node
W20260923 18:14:58.162117 15892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:58.162849 15886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:58.164078 15886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:58.165220 15886 hybrid_clock.cc:648] HybridClock initialized: now 1790187298165187 us; error 45 us; skew 500 ppm
Sep 23 18:14:58 dist-test-slave-wsx5 krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:58.168157 15886 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:14:58.169235 15886 webserver.cc:460] Webserver started at http://127.0.0.1:42981/ using document root <none> and password file <none>
I20260923 18:14:58.169821 15886 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:58.169891 15886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:58.170104 15886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:58.172540 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "a9ee850ea7944a09b33623e4a3909075"
format_stamp: "Formatted at 2026-09-23 18:14:58 on dist-test-slave-wsx5"
server_key: "4631f58e970359bcff864a1261594b9d"
server_key_iv: "37182ce39eb3f14dfa098ab4807936f9"
server_key_version: "encryptionkey@0"
I20260923 18:14:58.173365 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "a9ee850ea7944a09b33623e4a3909075"
format_stamp: "Formatted at 2026-09-23 18:14:58 on dist-test-slave-wsx5"
server_key: "4631f58e970359bcff864a1261594b9d"
server_key_iv: "37182ce39eb3f14dfa098ab4807936f9"
server_key_version: "encryptionkey@0"
I20260923 18:14:58.177690 15886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:14:58.180581 15901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:58.182071 15886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:58.182214 15886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "a9ee850ea7944a09b33623e4a3909075"
format_stamp: "Formatted at 2026-09-23 18:14:58 on dist-test-slave-wsx5"
server_key: "4631f58e970359bcff864a1261594b9d"
server_key_iv: "37182ce39eb3f14dfa098ab4807936f9"
server_key_version: "encryptionkey@0"
I20260923 18:14:58.182332 15886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:58.206573 15886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:58.210167 15886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:58.210378 15886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:58.217948 15886 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40755
I20260923 18:14:58.217969 15955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40755 every 8 connection(s)
I20260923 18:14:58.218941 15886 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:14:58.222095 15956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:58.224901 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 15886
I20260923 18:14:58.225085 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:14:58.225379 27951 external_mini_cluster.cc:1428] Setting key 6c1bdfa4bd297396d5ac60384b7361b7
I20260923 18:14:58.228392 15956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075: Bootstrap starting.
I20260923 18:14:58.231019 15956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:58.231858 15956 log.cc:824] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:58.234133 15956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075: No bootstrap required, opened a new log
I20260923 18:14:58.234905 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:58.226899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46595 (local address 127.0.0.1:40755)
0923 18:14:58.227352 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:58.227367 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:58.227406 (+ 39us) server_negotiation.cc:407] Connection header received
0923 18:14:58.228091 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:58.228110 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:58.228495 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:58.228818 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:58.229215 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:58.230213 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:58.231394 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:58.231887 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:58.232040 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:58.233080 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:58.233085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:58.233099 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:58.233570 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:58.233630 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:58.233654 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:58.233762 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:58.233965 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 18:14:58.234105 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":113,"threads_started":1}
I20260923 18:14:58.238193 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:14:58.238090 15956 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ee850ea7944a09b33623e4a3909075" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } }
I20260923 18:14:58.238386 15956 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:58.238438 15956 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ee850ea7944a09b33623e4a3909075, State: Initialized, Role: FOLLOWER
I20260923 18:14:58.238904 15956 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ee850ea7944a09b33623e4a3909075" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } }
I20260923 18:14:58.239066 15956 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:14:58.239130 15956 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:14:58.239229 15956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:58.240456 15956 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ee850ea7944a09b33623e4a3909075" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } }
I20260923 18:14:58.240952 15956 leader_election.cc:302] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9ee850ea7944a09b33623e4a3909075; no voters:
I20260923 18:14:58.241390 15956 leader_election.cc:288] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:14:58.241483 15966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:58.241703 15966 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [term 1 LEADER]: Becoming Leader. State: Replica: a9ee850ea7944a09b33623e4a3909075, State: Running, Role: LEADER
I20260923 18:14:58.242079 15966 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ee850ea7944a09b33623e4a3909075" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } }
I20260923 18:14:58.242808 15956 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:14:58.243779 15968 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9ee850ea7944a09b33623e4a3909075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ee850ea7944a09b33623e4a3909075" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } } }
I20260923 18:14:58.243909 15968 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:58.244601 15967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9ee850ea7944a09b33623e4a3909075. Latest consensus state: current_term: 1 leader_uuid: "a9ee850ea7944a09b33623e4a3909075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ee850ea7944a09b33623e4a3909075" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40755 } } }
I20260923 18:14:58.244693 15967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075 [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:58.245029 15975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:14:58.245088 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:58.240533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57797 (local address 127.0.0.1:40755)
0923 18:14:58.240738 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:58.240743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:58.240869 (+ 126us) server_negotiation.cc:407] Connection header received
0923 18:14:58.240974 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:58.240977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:58.241029 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:58.241150 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:58.241462 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:58.242169 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:58.243014 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:14:58.243274 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:14:58.243364 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:14:58.244392 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:58.244396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:58.244398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:58.244413 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:58.244442 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:58.244455 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:58.244541 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:58.244894 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 18:14:58.244932 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260923 18:14:58.247817 15975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:14:58.252995 15975 catalog_manager.cc:1347] Generated new cluster ID: f02f3c6a942246f785042361f54a7558
I20260923 18:14:58.253072 15975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:14:58.261875 15975 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:14:58.262866 15975 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:14:58.269418 15975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9ee850ea7944a09b33623e4a3909075: Generated new TSK 0
I20260923 18:14:58.270147 15975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:14:58.281301 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 15886
W20260923 18:14:58.286044 15871 connection.cc:535] client connection to 127.0.0.1:40755 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40755): Connection reset by peer (error 104)
2026-09-23T18:14:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:14:58 dist-test-slave-wsx5 krb5kdc[15987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:14:58 dist-test-slave-wsx5 krb5kdc[15987](info): set up 2 sockets
Sep 23 18:14:58 dist-test-slave-wsx5 krb5kdc[15987](info): commencing operation
krb5kdc: starting...
W20260923 18:14:59.235306 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:14:59 dist-test-slave-wsx5 krb5kdc[15987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:14:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:14:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:14:59.356124 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:14:59.469826 16000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:14:59.470145 16000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:14:59.470189 16000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:14:59.474391 16000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:14:59.474469 16000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:14:59.474488 16000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:14:59.474504 16000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:14:59.474519 16000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:14:59.480297 16000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34043
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:14:59.481576 16000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:14:59.482656 16000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:14:59.491703 16005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:59.491751 16006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:14:59.491748 16008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:14:59.491971 16000 server_base.cc:1034] running on GCE node
I20260923 18:14:59.492604 16000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:14:59.493896 16000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:14:59.495074 16000 hybrid_clock.cc:648] HybridClock initialized: now 1790187299495053 us; error 54 us; skew 500 ppm
Sep 23 18:14:59 dist-test-slave-wsx5 krb5kdc[15987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:14:59.497920 16000 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:14:59.498894 16000 webserver.cc:460] Webserver started at http://127.0.0.1:39963/ using document root <none> and password file <none>
I20260923 18:14:59.499436 16000 fs_manager.cc:359] Metadata directory not provided
I20260923 18:14:59.499498 16000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:14:59.499691 16000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:14:59.501853 16000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "da8b0f11a3b74303b7bf5aa8af3fb0aa"
format_stamp: "Formatted at 2026-09-23 18:14:59 on dist-test-slave-wsx5"
server_key: "bc9258548b6b62adc3d6a254a5843cd8"
server_key_iv: "f5573cdced3886e05de4f7a2a71b9de6"
server_key_version: "encryptionkey@0"
I20260923 18:14:59.502413 16000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "da8b0f11a3b74303b7bf5aa8af3fb0aa"
format_stamp: "Formatted at 2026-09-23 18:14:59 on dist-test-slave-wsx5"
server_key: "bc9258548b6b62adc3d6a254a5843cd8"
server_key_iv: "f5573cdced3886e05de4f7a2a71b9de6"
server_key_version: "encryptionkey@0"
I20260923 18:14:59.506299 16000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:14:59.509155 16015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:14:59.510689 16000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:14:59.510874 16000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "da8b0f11a3b74303b7bf5aa8af3fb0aa"
format_stamp: "Formatted at 2026-09-23 18:14:59 on dist-test-slave-wsx5"
server_key: "bc9258548b6b62adc3d6a254a5843cd8"
server_key_iv: "f5573cdced3886e05de4f7a2a71b9de6"
server_key_version: "encryptionkey@0"
I20260923 18:14:59.511014 16000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:14:59.515803 16000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:14:59.519021 16000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:14:59.519187 16000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:14:59.526079 16000 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34043
I20260923 18:14:59.526144 16067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34043 every 8 connection(s)
I20260923 18:14:59.527055 16000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:14:59.529983 16068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:14:59.532574 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 16000
I20260923 18:14:59.532733 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:14:59.533034 27951 external_mini_cluster.cc:1428] Setting key 96b8727ea1414887e9fc887e8fae16f2
I20260923 18:14:59.536309 16068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa: Bootstrap starting.
I20260923 18:14:59.538549 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:59.534471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40429 (local address 127.0.0.1:34043)
0923 18:14:59.534914 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:59.534929 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:59.534965 (+ 36us) server_negotiation.cc:407] Connection header received
0923 18:14:59.535620 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:59.535638 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:59.535944 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:59.536242 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:59.537252 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:59.537257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:59.537297 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:59.537309 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:59.537368 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:59.537393 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:59.537506 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:59.537619 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 18:14:59.537765 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":142,"threads_started":1}
I20260923 18:14:59.538836 16068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa: Neither blocks nor log segments found. Creating new log.
I20260923 18:14:59.539558 16068 log.cc:824] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa: Log is configured to *not* fsync() on all Append() calls
I20260923 18:14:59.541496 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:14:59.541610 16068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa: No bootstrap required, opened a new log
I20260923 18:14:59.545554 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:14:59.543801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49605 (local address 127.0.0.1:34043)
0923 18:14:59.543931 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 18:14:59.543936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:14:59.544240 (+ 304us) server_negotiation.cc:407] Connection header received
0923 18:14:59.544362 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:14:59.544365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:14:59.544404 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:14:59.544488 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:14:59.545159 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:14:59.545163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:14:59.545166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:14:59.545179 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:14:59.545205 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:14:59.545218 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:14:59.545309 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:14:59.545393 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 18:14:59.545433 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:14:59.545967 16068 raft_consensus.cc:348] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b0f11a3b74303b7bf5aa8af3fb0aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34043 } }
I20260923 18:14:59.546206 16068 raft_consensus.cc:374] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:14:59.546260 16068 raft_consensus.cc:729] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8b0f11a3b74303b7bf5aa8af3fb0aa, State: Initialized, Role: FOLLOWER
I20260923 18:14:59.546689 16068 consensus_queue.cc:260] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b0f11a3b74303b7bf5aa8af3fb0aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34043 } }
I20260923 18:14:59.546833 16068 raft_consensus.cc:388] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:14:59.546898 16068 raft_consensus.cc:482] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:14:59.546998 16068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:14:59.547912 16068 raft_consensus.cc:504] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b0f11a3b74303b7bf5aa8af3fb0aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34043 } }
I20260923 18:14:59.548251 16068 leader_election.cc:302] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da8b0f11a3b74303b7bf5aa8af3fb0aa; no voters:
I20260923 18:14:59.548553 16068 leader_election.cc:288] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:14:59.548700 16078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:14:59.548951 16078 raft_consensus.cc:686] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [term 1 LEADER]: Becoming Leader. State: Replica: da8b0f11a3b74303b7bf5aa8af3fb0aa, State: Running, Role: LEADER
I20260923 18:14:59.549304 16078 consensus_queue.cc:237] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b0f11a3b74303b7bf5aa8af3fb0aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34043 } }
I20260923 18:14:59.549978 16068 sys_catalog.cc:564] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:14:59.550853 16079 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da8b0f11a3b74303b7bf5aa8af3fb0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b0f11a3b74303b7bf5aa8af3fb0aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34043 } } }
I20260923 18:14:59.551138 16080 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader da8b0f11a3b74303b7bf5aa8af3fb0aa. Latest consensus state: current_term: 1 leader_uuid: "da8b0f11a3b74303b7bf5aa8af3fb0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b0f11a3b74303b7bf5aa8af3fb0aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34043 } } }
I20260923 18:14:59.551229 16079 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:59.551314 16080 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa [sys.catalog]: This master's current role is: LEADER
I20260923 18:14:59.551759 16088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:14:59.554675 16088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:14:59.560892 16088 catalog_manager.cc:1347] Generated new cluster ID: 03f0b0a7449e4cd1912f4fb1b4dc0924
I20260923 18:14:59.560985 16088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:14:59.577692 16088 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:14:59.578028 16088 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:14:59.582953 16088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da8b0f11a3b74303b7bf5aa8af3fb0aa: Generated new TSK 0
I20260923 18:14:59.583649 16088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:14:59.593863 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 16000
2026-09-23T18:14:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:14:59 dist-test-slave-wsx5 krb5kdc[16099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:14:59 dist-test-slave-wsx5 krb5kdc[16099](info): set up 2 sockets
Sep 23 18:14:59 dist-test-slave-wsx5 krb5kdc[16099](info): commencing operation
krb5kdc: starting...
W20260923 18:15:00.555678 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:00 dist-test-slave-wsx5 krb5kdc[16099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:15:00.689075 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:00.808704 16112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:00.809021 16112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:00.809067 16112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:00.813367 16112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:00.813457 16112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:00.813476 16112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:00.813493 16112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:00.813506 16112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:00.819171 16112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:00.820327 16112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:00.821414 16112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:00.830217 16120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:00.830224 16117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:00.830243 16118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:00.830437 16112 server_base.cc:1034] running on GCE node
I20260923 18:15:00.831139 16112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:00.832325 16112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:00.833500 16112 hybrid_clock.cc:648] HybridClock initialized: now 1790187300833473 us; error 43 us; skew 500 ppm
Sep 23 18:15:00 dist-test-slave-wsx5 krb5kdc[16099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:00.843003 16112 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:15:00.844101 16112 webserver.cc:460] Webserver started at http://127.0.0.1:41981/ using document root <none> and password file <none>
I20260923 18:15:00.844712 16112 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:00.844787 16112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:00.844995 16112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:00.847220 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "2031ace3661344d4a634661c582c967b"
format_stamp: "Formatted at 2026-09-23 18:15:00 on dist-test-slave-wsx5"
server_key: "c394f4d2d2fd6fd7a2853e5e2ecad9a4"
server_key_iv: "6edd4d3ccf5875c1c5df198cf7e4c086"
server_key_version: "encryptionkey@0"
I20260923 18:15:00.847815 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "2031ace3661344d4a634661c582c967b"
format_stamp: "Formatted at 2026-09-23 18:15:00 on dist-test-slave-wsx5"
server_key: "c394f4d2d2fd6fd7a2853e5e2ecad9a4"
server_key_iv: "6edd4d3ccf5875c1c5df198cf7e4c086"
server_key_version: "encryptionkey@0"
I20260923 18:15:00.852129 16112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:15:00.855049 16127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:00.856573 16112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:15:00.856720 16112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "2031ace3661344d4a634661c582c967b"
format_stamp: "Formatted at 2026-09-23 18:15:00 on dist-test-slave-wsx5"
server_key: "c394f4d2d2fd6fd7a2853e5e2ecad9a4"
server_key_iv: "6edd4d3ccf5875c1c5df198cf7e4c086"
server_key_version: "encryptionkey@0"
I20260923 18:15:00.856842 16112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:00.900027 16112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:00.903697 16112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:00.903908 16112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:00.911324 16112 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39355
I20260923 18:15:00.911379 16179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39355 every 8 connection(s)
I20260923 18:15:00.912320 16112 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:00.915248 16180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:00.916921 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 16112
I20260923 18:15:00.917088 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:00.917373 27951 external_mini_cluster.cc:1428] Setting key e9bedef8f8d745fd88af147404e0f38e
I20260923 18:15:00.921485 16180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b: Bootstrap starting.
I20260923 18:15:00.923612 16180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:00.924348 16180 log.cc:824] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:15:00 dist-test-slave-wsx5 krb5kdc[16099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 18:15:00.926494 16180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b: No bootstrap required, opened a new log
I20260923 18:15:00.930089 16180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031ace3661344d4a634661c582c967b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39355 } }
I20260923 18:15:00.930374 16180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:00.930434 16180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2031ace3661344d4a634661c582c967b, State: Initialized, Role: FOLLOWER
I20260923 18:15:00.930888 16180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031ace3661344d4a634661c582c967b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39355 } }
I20260923 18:15:00.931013 16180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:00.931073 16180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:00.931164 16180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:00.931198 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:00.918726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43027 (local address 127.0.0.1:39355)
0923 18:15:00.919185 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:00.919200 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:00.919237 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:15:00.919905 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:00.919923 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:00.920274 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:00.920608 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:00.921455 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:00.922460 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:00.923309 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:00.923812 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:00.923969 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:00.926665 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:00.926670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:00.926684 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:00.926710 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:00.928266 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:00.928693 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:00.928698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:00.928702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:00.928872 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:00.929187 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:00.929190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:00.929191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:00.929589 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:00.929781 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:00.930054 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 18:15:00.930345 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":152,"threads_started":1}
I20260923 18:15:00.932097 16180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031ace3661344d4a634661c582c967b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39355 } }
I20260923 18:15:00.932432 16180 leader_election.cc:302] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2031ace3661344d4a634661c582c967b; no voters:
I20260923 18:15:00.932739 16180 leader_election.cc:288] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:00.932852 16185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:00.933066 16185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [term 1 LEADER]: Becoming Leader. State: Replica: 2031ace3661344d4a634661c582c967b, State: Running, Role: LEADER
I20260923 18:15:00.933478 16185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031ace3661344d4a634661c582c967b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39355 } }
I20260923 18:15:00.934088 16180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:00.934340 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:00.935235 16187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2031ace3661344d4a634661c582c967b. Latest consensus state: current_term: 1 leader_uuid: "2031ace3661344d4a634661c582c967b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031ace3661344d4a634661c582c967b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39355 } } }
I20260923 18:15:00.935350 16187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:00.935704 16194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:00.936326 16186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2031ace3661344d4a634661c582c967b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2031ace3661344d4a634661c582c967b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39355 } } }
I20260923 18:15:00.936458 16186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:00.939014 16194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:00.945859 16194 catalog_manager.cc:1347] Generated new cluster ID: fd012772ddbd4a8889a4571aa8273c82
I20260923 18:15:00.945947 16194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:00.949627 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:00.937047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54967 (local address 127.0.0.1:39355)
0923 18:15:00.937173 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:00.937178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:00.937671 (+ 493us) server_negotiation.cc:407] Connection header received
0923 18:15:00.937810 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:00.937813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:00.937863 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:00.937948 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:00.938907 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:00.939699 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:00.941042 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:00.941362 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:00.941434 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:00.942876 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:00.942880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:00.942883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:00.942918 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:00.944607 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:00.945022 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:00.945026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:00.945028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:00.945110 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:00.945384 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:00.945387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:00.945389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:00.945618 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:00.945737 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:00.949296 (+ 3559us) server_negotiation.cc:299] Negotiation successful
0923 18:15:00.949443 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 18:15:00.995903 16194 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:00.996927 16194 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:01.008404 16194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2031ace3661344d4a634661c582c967b: Generated new TSK 0
I20260923 18:15:01.009212 16194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:01.019984 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 16112
2026-09-23T18:15:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:01 dist-test-slave-wsx5 krb5kdc[16211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:01 dist-test-slave-wsx5 krb5kdc[16211](info): set up 2 sockets
Sep 23 18:15:01 dist-test-slave-wsx5 krb5kdc[16211](info): commencing operation
krb5kdc: starting...
W20260923 18:15:01.991858 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:02 dist-test-slave-wsx5 krb5kdc[16211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:15:02.117743 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:02.234606 16224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:02.234905 16224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:02.234949 16224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:02.239113 16224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:02.239197 16224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:02.239215 16224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:02.239233 16224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:02.239246 16224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:02.244894 16224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:02.246013 16224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:02.246995 16224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:15:02.255765 16224 server_base.cc:1034] running on GCE node
W20260923 18:15:02.255717 16232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:02.255722 16229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:02.255731 16230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:02.256613 16224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:02.257861 16224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:02.259056 16224 hybrid_clock.cc:648] HybridClock initialized: now 1790187302259023 us; error 48 us; skew 500 ppm
Sep 23 18:15:02 dist-test-slave-wsx5 krb5kdc[16211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:02.261951 16224 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:15:02.262974 16224 webserver.cc:460] Webserver started at http://127.0.0.1:35571/ using document root <none> and password file <none>
I20260923 18:15:02.263532 16224 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:02.263599 16224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:02.263801 16224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:02.265996 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "62b65e73f722441c8df720554d2f1ee0"
format_stamp: "Formatted at 2026-09-23 18:15:02 on dist-test-slave-wsx5"
server_key: "601adb9d0e72f1649299c4688bcc1a94"
server_key_iv: "5a37b87efbef3425e84e14292d79a341"
server_key_version: "encryptionkey@0"
I20260923 18:15:02.266573 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "62b65e73f722441c8df720554d2f1ee0"
format_stamp: "Formatted at 2026-09-23 18:15:02 on dist-test-slave-wsx5"
server_key: "601adb9d0e72f1649299c4688bcc1a94"
server_key_iv: "5a37b87efbef3425e84e14292d79a341"
server_key_version: "encryptionkey@0"
I20260923 18:15:02.270498 16224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:15:02.273214 16239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:02.274523 16224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:15:02.274660 16224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "62b65e73f722441c8df720554d2f1ee0"
format_stamp: "Formatted at 2026-09-23 18:15:02 on dist-test-slave-wsx5"
server_key: "601adb9d0e72f1649299c4688bcc1a94"
server_key_iv: "5a37b87efbef3425e84e14292d79a341"
server_key_version: "encryptionkey@0"
I20260923 18:15:02.274772 16224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:02.310006 16224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:02.313772 16224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:02.313985 16224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:02.321609 16224 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35939
I20260923 18:15:02.321668 16291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35939 every 8 connection(s)
I20260923 18:15:02.323572 16224 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:02.325134 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 16224
I20260923 18:15:02.325275 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:02.325570 27951 external_mini_cluster.cc:1428] Setting key 4a30f1b72458db4eb8b3ee42a1e630be
I20260923 18:15:02.327108 16292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:02.333572 16292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0: Bootstrap starting.
I20260923 18:15:02.335352 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:02.327151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46915 (local address 127.0.0.1:35939)
0923 18:15:02.327651 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:02.327667 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:02.327728 (+ 61us) server_negotiation.cc:407] Connection header received
0923 18:15:02.328448 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:02.328468 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:02.328843 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:02.329330 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:02.329794 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:02.330949 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:02.331903 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:02.332402 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:02.332556 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:02.333549 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:02.333554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:02.333568 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:02.334039 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:02.334106 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:02.334132 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:02.334225 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:02.334385 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 18:15:02.334608 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":159,"threads_started":1}
I20260923 18:15:02.336035 16292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:02.336910 16292 log.cc:824] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:02.338373 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:02.339171 16292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0: No bootstrap required, opened a new log
I20260923 18:15:02.343024 16292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b65e73f722441c8df720554d2f1ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260923 18:15:02.343294 16292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:02.343356 16292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b65e73f722441c8df720554d2f1ee0, State: Initialized, Role: FOLLOWER
I20260923 18:15:02.343848 16292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b65e73f722441c8df720554d2f1ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260923 18:15:02.343997 16292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:02.344099 16292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:02.344233 16292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:02.344260 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:02.340415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40349 (local address 127.0.0.1:35939)
0923 18:15:02.340545 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:02.340550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:02.340660 (+ 110us) server_negotiation.cc:407] Connection header received
0923 18:15:02.340760 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:02.340763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:02.340813 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:02.340903 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:02.341257 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:02.341798 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:02.342613 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:02.342868 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:02.342967 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:02.343818 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:02.343821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:02.343824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:02.343839 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:02.343867 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:02.343878 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:02.343944 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:02.344058 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 18:15:02.344126 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:15:02.345472 16292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b65e73f722441c8df720554d2f1ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260923 18:15:02.345834 16292 leader_election.cc:302] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62b65e73f722441c8df720554d2f1ee0; no voters:
I20260923 18:15:02.346144 16292 leader_election.cc:288] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:02.346274 16302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:02.346493 16302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [term 1 LEADER]: Becoming Leader. State: Replica: 62b65e73f722441c8df720554d2f1ee0, State: Running, Role: LEADER
I20260923 18:15:02.346921 16302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b65e73f722441c8df720554d2f1ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260923 18:15:02.347438 16292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:02.348568 16303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62b65e73f722441c8df720554d2f1ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b65e73f722441c8df720554d2f1ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } } }
I20260923 18:15:02.348687 16303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:02.349081 16311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:02.349023 16304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62b65e73f722441c8df720554d2f1ee0. Latest consensus state: current_term: 1 leader_uuid: "62b65e73f722441c8df720554d2f1ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b65e73f722441c8df720554d2f1ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } } }
I20260923 18:15:02.349258 16304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:02.351939 16311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:02.358448 16311 catalog_manager.cc:1347] Generated new cluster ID: 83ad744351524a609612758426572a4d
I20260923 18:15:02.358531 16311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:02.366112 16311 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:02.367126 16311 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:02.372663 16311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62b65e73f722441c8df720554d2f1ee0: Generated new TSK 0
I20260923 18:15:02.373585 16311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:02.379614 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 16224
2026-09-23T18:15:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:02 dist-test-slave-wsx5 krb5kdc[16323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:02 dist-test-slave-wsx5 krb5kdc[16323](info): set up 2 sockets
Sep 23 18:15:02 dist-test-slave-wsx5 krb5kdc[16323](info): commencing operation
krb5kdc: starting...
W20260923 18:15:03.351248 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:03 dist-test-slave-wsx5 krb5kdc[16323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:15:03.473089 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:03.588238 16336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:03.588594 16336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:03.588639 16336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:03.592815 16336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:03.592901 16336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:03.592919 16336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:03.592934 16336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:03.592949 16336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:03.598600 16336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:03.599712 16336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:03.600705 16336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:03.609818 16342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:03.609997 16344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:03.609922 16341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:03.610390 16336 server_base.cc:1034] running on GCE node
I20260923 18:15:03.611233 16336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:03.612702 16336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:03.613955 16336 hybrid_clock.cc:648] HybridClock initialized: now 1790187303613928 us; error 49 us; skew 500 ppm
Sep 23 18:15:03 dist-test-slave-wsx5 krb5kdc[16323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:03.616990 16336 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:15:03.618129 16336 webserver.cc:460] Webserver started at http://127.0.0.1:33771/ using document root <none> and password file <none>
I20260923 18:15:03.618745 16336 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:03.618858 16336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:03.619087 16336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:03.621340 16336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "8910781847b84e2db5ad456f0cd48b80"
format_stamp: "Formatted at 2026-09-23 18:15:03 on dist-test-slave-wsx5"
server_key: "341fb25fc9eb10ade37cdd11147f76e5"
server_key_iv: "98032010460cd0057b338f813159f6c0"
server_key_version: "encryptionkey@0"
I20260923 18:15:03.621953 16336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "8910781847b84e2db5ad456f0cd48b80"
format_stamp: "Formatted at 2026-09-23 18:15:03 on dist-test-slave-wsx5"
server_key: "341fb25fc9eb10ade37cdd11147f76e5"
server_key_iv: "98032010460cd0057b338f813159f6c0"
server_key_version: "encryptionkey@0"
I20260923 18:15:03.625886 16336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:15:03.628528 16351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:03.629971 16336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 18:15:03.630103 16336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "8910781847b84e2db5ad456f0cd48b80"
format_stamp: "Formatted at 2026-09-23 18:15:03 on dist-test-slave-wsx5"
server_key: "341fb25fc9eb10ade37cdd11147f76e5"
server_key_iv: "98032010460cd0057b338f813159f6c0"
server_key_version: "encryptionkey@0"
I20260923 18:15:03.630213 16336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:03.634986 16336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:03.638167 16336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:03.638345 16336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:03.645368 16336 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35811
I20260923 18:15:03.645396 16404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35811 every 8 connection(s)
I20260923 18:15:03.646348 16336 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:03.649256 16405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:03.649479 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 16336
I20260923 18:15:03.649597 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:03.649849 27951 external_mini_cluster.cc:1428] Setting key 1e359875e3c13a87c956f73b3e555ccf
I20260923 18:15:03.655748 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:03.651284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45719 (local address 127.0.0.1:35811)
0923 18:15:03.651737 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:03.651758 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:03.651803 (+ 45us) server_negotiation.cc:407] Connection header received
0923 18:15:03.652581 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:03.652608 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:03.653017 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:03.653460 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:03.654266 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:03.654272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:03.654288 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:03.654301 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:03.654368 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:03.654395 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:03.654543 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:03.654612 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 18:15:03.654772 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":122,"threads_started":1}
I20260923 18:15:03.656726 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80: Bootstrap starting.
I20260923 18:15:03.658926 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:03.659485 16405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:03.660424 16405 log.cc:824] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:03.662775 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80: No bootstrap required, opened a new log
I20260923 18:15:03.662916 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:03.661285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45519 (local address 127.0.0.1:35811)
0923 18:15:03.661419 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:03.661424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:03.661586 (+ 162us) server_negotiation.cc:407] Connection header received
0923 18:15:03.661687 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:03.661690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:03.661726 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:03.661810 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:03.662454 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:03.662458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:03.662461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:03.662472 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:03.662502 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:03.662515 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:03.662606 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:03.662728 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 18:15:03.662768 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 18:15:03.666548 16405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8910781847b84e2db5ad456f0cd48b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260923 18:15:03.666821 16405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:03.666864 16405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8910781847b84e2db5ad456f0cd48b80, State: Initialized, Role: FOLLOWER
I20260923 18:15:03.667268 16405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8910781847b84e2db5ad456f0cd48b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260923 18:15:03.667488 16405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:03.667598 16405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:03.667717 16405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:03.668699 16405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8910781847b84e2db5ad456f0cd48b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260923 18:15:03.669055 16405 leader_election.cc:302] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8910781847b84e2db5ad456f0cd48b80; no voters:
I20260923 18:15:03.669425 16405 leader_election.cc:288] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:03.669543 16415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:03.669765 16415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [term 1 LEADER]: Becoming Leader. State: Replica: 8910781847b84e2db5ad456f0cd48b80, State: Running, Role: LEADER
I20260923 18:15:03.670128 16415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8910781847b84e2db5ad456f0cd48b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260923 18:15:03.670728 16405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:03.671948 16416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8910781847b84e2db5ad456f0cd48b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8910781847b84e2db5ad456f0cd48b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } } }
I20260923 18:15:03.671983 16417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8910781847b84e2db5ad456f0cd48b80. Latest consensus state: current_term: 1 leader_uuid: "8910781847b84e2db5ad456f0cd48b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8910781847b84e2db5ad456f0cd48b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } } }
I20260923 18:15:03.672075 16416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:03.672076 16417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:03.672498 16425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:03.675257 16425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:03.680747 16425 catalog_manager.cc:1347] Generated new cluster ID: 8199020593c041dfb66d1ad07608f85b
I20260923 18:15:03.680836 16425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:03.696278 16425 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:03.696693 16425 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:03.705449 16425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8910781847b84e2db5ad456f0cd48b80: Generated new TSK 0
I20260923 18:15:03.706413 16425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:03.713078 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 16336
2026-09-23T18:15:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:03 dist-test-slave-wsx5 krb5kdc[16436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:03 dist-test-slave-wsx5 krb5kdc[16436](info): set up 2 sockets
Sep 23 18:15:03 dist-test-slave-wsx5 krb5kdc[16436](info): commencing operation
krb5kdc: starting...
W20260923 18:15:04.675491 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:04 dist-test-slave-wsx5 krb5kdc[16436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:15:04.804965 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:04.925254 16450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:04.925549 16450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:04.925592 16450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:04.929629 16450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:04.929700 16450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:04.929718 16450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:04.929734 16450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:04.929749 16450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:04.935460 16450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:04.936580 16450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:04.937623 16450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:04.946868 16455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:04.946879 16458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:04.946888 16456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:04.947063 16450 server_base.cc:1034] running on GCE node
I20260923 18:15:04.947942 16450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:04.949306 16450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:04.950495 16450 hybrid_clock.cc:648] HybridClock initialized: now 1790187304950462 us; error 48 us; skew 500 ppm
Sep 23 18:15:04 dist-test-slave-wsx5 krb5kdc[16436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:04.953845 16450 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:15:04.954833 16450 webserver.cc:460] Webserver started at http://127.0.0.1:39475/ using document root <none> and password file <none>
I20260923 18:15:04.955394 16450 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:04.955459 16450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:04.955633 16450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:04.957919 16450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "0d330c829a7e4f43b661acc66f0ac93e"
format_stamp: "Formatted at 2026-09-23 18:15:04 on dist-test-slave-wsx5"
server_key: "2122a56de6031a85a38dae6a46f98269"
server_key_iv: "9dfc91e3eaf103e2c56410d368b9eb1f"
server_key_version: "encryptionkey@0"
I20260923 18:15:04.958495 16450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "0d330c829a7e4f43b661acc66f0ac93e"
format_stamp: "Formatted at 2026-09-23 18:15:04 on dist-test-slave-wsx5"
server_key: "2122a56de6031a85a38dae6a46f98269"
server_key_iv: "9dfc91e3eaf103e2c56410d368b9eb1f"
server_key_version: "encryptionkey@0"
I20260923 18:15:04.962642 16450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:15:04.965444 16465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:04.966908 16450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:15:04.967032 16450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "0d330c829a7e4f43b661acc66f0ac93e"
format_stamp: "Formatted at 2026-09-23 18:15:04 on dist-test-slave-wsx5"
server_key: "2122a56de6031a85a38dae6a46f98269"
server_key_iv: "9dfc91e3eaf103e2c56410d368b9eb1f"
server_key_version: "encryptionkey@0"
I20260923 18:15:04.967136 16450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:05.003126 16450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:05.006461 16450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:05.006628 16450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:05.013967 16450 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37065
I20260923 18:15:05.014037 16518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37065 every 8 connection(s)
I20260923 18:15:05.015025 16450 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:05.018141 16519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:05.023075 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 16450
I20260923 18:15:05.023223 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:05.023512 27951 external_mini_cluster.cc:1428] Setting key 0b088f47cc2930af89a784406cd3a843
I20260923 18:15:05.024897 16519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e: Bootstrap starting.
I20260923 18:15:05.027755 16519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:05.028689 16519 log.cc:824] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:05.030961 16519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e: No bootstrap required, opened a new log
Sep 23 18:15:05 dist-test-slave-wsx5 krb5kdc[16436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 18:15:05.035188 16519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d330c829a7e4f43b661acc66f0ac93e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } }
I20260923 18:15:05.035506 16519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:05.035557 16519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d330c829a7e4f43b661acc66f0ac93e, State: Initialized, Role: FOLLOWER
I20260923 18:15:05.036077 16519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d330c829a7e4f43b661acc66f0ac93e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } }
I20260923 18:15:05.036232 16519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:05.036310 16519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:05.036408 16519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:05.037509 16519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d330c829a7e4f43b661acc66f0ac93e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } }
I20260923 18:15:05.037909 16519 leader_election.cc:302] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d330c829a7e4f43b661acc66f0ac93e; no voters:
I20260923 18:15:05.038300 16519 leader_election.cc:288] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:05.038378 16524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:05.038578 16524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [term 1 LEADER]: Becoming Leader. State: Replica: 0d330c829a7e4f43b661acc66f0ac93e, State: Running, Role: LEADER
I20260923 18:15:05.038904 16524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d330c829a7e4f43b661acc66f0ac93e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } }
I20260923 18:15:05.039657 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:05.025090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:46081 (local address 127.0.0.1:37065)
0923 18:15:05.025567 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:05.025586 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:05.025628 (+ 42us) server_negotiation.cc:407] Connection header received
0923 18:15:05.026530 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:05.026562 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:05.027033 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:05.027412 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:05.028148 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:05.029521 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:05.030433 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:05.031138 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:05.031310 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:05.033883 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:05.033890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:05.033910 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:05.033945 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:05.035913 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:05.036504 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:05.036511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:05.036518 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:05.036656 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:05.037050 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:05.037054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:05.037056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:05.037674 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:05.037900 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:05.038556 (+ 656us) server_negotiation.cc:299] Negotiation successful
0923 18:15:05.038795 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":130,"threads_started":1}
I20260923 18:15:05.039707 16519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:05.040167 16524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d330c829a7e4f43b661acc66f0ac93e. Latest consensus state: current_term: 1 leader_uuid: "0d330c829a7e4f43b661acc66f0ac93e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d330c829a7e4f43b661acc66f0ac93e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } } }
I20260923 18:15:05.040282 16524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:05.040474 16524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d330c829a7e4f43b661acc66f0ac93e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d330c829a7e4f43b661acc66f0ac93e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } } }
I20260923 18:15:05.040551 16524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:05.040825 16528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:05.043205 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:05.045148 16528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:05.052104 16528 catalog_manager.cc:1347] Generated new cluster ID: fd658982e0594764bb26d3b96182eecd
I20260923 18:15:05.052187 16528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:05.055857 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:05.045836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:48423 (local address 127.0.0.1:37065)
0923 18:15:05.045952 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:05.045957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:05.046383 (+ 426us) server_negotiation.cc:407] Connection header received
0923 18:15:05.046511 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:05.046514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:05.046570 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:05.046666 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:05.047516 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:05.048330 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:05.049514 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:05.049908 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:05.050013 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:05.051896 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:05.051900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:05.051903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:05.051932 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:05.053673 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:05.054302 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:05.054305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:05.054307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:05.054401 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:05.054800 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:05.054803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:05.054805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:05.055023 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:05.055139 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:05.055581 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 18:15:05.055687 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 18:15:05.067132 16528 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:05.068117 16528 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:05.076849 16528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d330c829a7e4f43b661acc66f0ac93e: Generated new TSK 0
I20260923 18:15:05.077811 16528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:05.085316 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 16450
W20260923 18:15:05.090114 16434 connection.cc:535] client connection to 127.0.0.1:37065 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37065): Connection reset by peer (error 104)
2026-09-23T18:15:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:05 dist-test-slave-wsx5 krb5kdc[16550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:05 dist-test-slave-wsx5 krb5kdc[16550](info): set up 2 sockets
Sep 23 18:15:05 dist-test-slave-wsx5 krb5kdc[16550](info): commencing operation
krb5kdc: starting...
W20260923 18:15:06.051373 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:06 dist-test-slave-wsx5 krb5kdc[16550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:15:06.176780 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:06.294277 16563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:06.294593 16563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:06.294638 16563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:06.298902 16563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:06.298977 16563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:06.298995 16563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:06.299011 16563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:06.299026 16563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:06.304804 16563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:06.306043 16563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:06.307060 16563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:06.315778 16569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:06.315827 16571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:06.315778 16568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:06.316143 16563 server_base.cc:1034] running on GCE node
I20260923 18:15:06.316689 16563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:06.317915 16563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:06.319092 16563 hybrid_clock.cc:648] HybridClock initialized: now 1790187306319074 us; error 43 us; skew 500 ppm
Sep 23 18:15:06 dist-test-slave-wsx5 krb5kdc[16550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:06.332950 16563 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:15:06.334009 16563 webserver.cc:460] Webserver started at http://127.0.0.1:39229/ using document root <none> and password file <none>
I20260923 18:15:06.334581 16563 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:06.334648 16563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:06.334854 16563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:06.336953 16563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "054370ef5d64485eb7fd813e8b4d50ca"
format_stamp: "Formatted at 2026-09-23 18:15:06 on dist-test-slave-wsx5"
server_key: "70b68511f71ae2b0ff6e25c8d572d67a"
server_key_iv: "e0bbf798464a8e70df97c532471ffe3c"
server_key_version: "encryptionkey@0"
I20260923 18:15:06.337560 16563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "054370ef5d64485eb7fd813e8b4d50ca"
format_stamp: "Formatted at 2026-09-23 18:15:06 on dist-test-slave-wsx5"
server_key: "70b68511f71ae2b0ff6e25c8d572d67a"
server_key_iv: "e0bbf798464a8e70df97c532471ffe3c"
server_key_version: "encryptionkey@0"
I20260923 18:15:06.341789 16563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 18:15:06.344331 16578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:06.345697 16563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:15:06.345822 16563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "054370ef5d64485eb7fd813e8b4d50ca"
format_stamp: "Formatted at 2026-09-23 18:15:06 on dist-test-slave-wsx5"
server_key: "70b68511f71ae2b0ff6e25c8d572d67a"
server_key_iv: "e0bbf798464a8e70df97c532471ffe3c"
server_key_version: "encryptionkey@0"
I20260923 18:15:06.345948 16563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:06.362888 16563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:06.366837 16563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:06.367031 16563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:06.374302 16563 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38547
I20260923 18:15:06.374321 16630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38547 every 8 connection(s)
I20260923 18:15:06.375288 16563 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:06.378206 16631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:06.383781 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 16563
I20260923 18:15:06.383936 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:06.384086 16631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca: Bootstrap starting.
I20260923 18:15:06.384202 27951 external_mini_cluster.cc:1428] Setting key 5a9caf3bdd30c89ad5440fe2ff58fc50
I20260923 18:15:06.386794 16631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:06.387542 16631 log.cc:824] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:06.389559 16631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca: No bootstrap required, opened a new log
I20260923 18:15:06.393724 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:06.385728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:56183 (local address 127.0.0.1:38547)
0923 18:15:06.386190 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:06.386205 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:06.386242 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:15:06.386953 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:06.386979 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:06.387356 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:06.387683 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:06.388112 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:06.389089 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:06.390023 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:06.390504 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:06.390654 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:06.391778 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:06.391783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:06.391798 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:06.392255 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:06.392318 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:06.392343 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:06.392461 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:06.392660 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 18:15:06.392821 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":141,"threads_started":1}
I20260923 18:15:06.393707 16631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054370ef5d64485eb7fd813e8b4d50ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260923 18:15:06.394014 16631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:06.394074 16631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054370ef5d64485eb7fd813e8b4d50ca, State: Initialized, Role: FOLLOWER
I20260923 18:15:06.394541 16631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054370ef5d64485eb7fd813e8b4d50ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260923 18:15:06.394685 16631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:06.394735 16631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:06.394819 16631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:06.395818 16631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054370ef5d64485eb7fd813e8b4d50ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260923 18:15:06.396162 16631 leader_election.cc:302] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 054370ef5d64485eb7fd813e8b4d50ca; no voters:
I20260923 18:15:06.396507 16631 leader_election.cc:288] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:06.396589 16636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:06.396778 16636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [term 1 LEADER]: Becoming Leader. State: Replica: 054370ef5d64485eb7fd813e8b4d50ca, State: Running, Role: LEADER
I20260923 18:15:06.396859 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:06.397092 16636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054370ef5d64485eb7fd813e8b4d50ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } }
I20260923 18:15:06.397908 16631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:06.398378 16637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 054370ef5d64485eb7fd813e8b4d50ca. Latest consensus state: current_term: 1 leader_uuid: "054370ef5d64485eb7fd813e8b4d50ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054370ef5d64485eb7fd813e8b4d50ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } } }
I20260923 18:15:06.398502 16637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:06.399124 16638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "054370ef5d64485eb7fd813e8b4d50ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054370ef5d64485eb7fd813e8b4d50ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38547 } } }
I20260923 18:15:06.399243 16638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:06.399183 16645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:06.403759 16645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:06.404996 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:06.399754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:35977 (local address 127.0.0.1:38547)
0923 18:15:06.399870 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:06.399874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:06.400356 (+ 482us) server_negotiation.cc:407] Connection header received
0923 18:15:06.400499 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:06.400503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:06.400555 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:06.400624 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:06.400939 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:06.401554 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:06.402756 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:06.403132 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:06.403227 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:06.404364 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:06.404368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:06.404371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:06.404388 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:06.404418 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:06.404433 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:06.404492 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:06.404805 (+ 313us) server_negotiation.cc:299] Negotiation successful
0923 18:15:06.404843 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 18:15:06.410459 16645 catalog_manager.cc:1347] Generated new cluster ID: 91c35596f0a04f0aaf2aea546cf9889d
I20260923 18:15:06.410548 16645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:06.430321 16645 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:06.431664 16645 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:06.438153 16645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 054370ef5d64485eb7fd813e8b4d50ca: Generated new TSK 0
I20260923 18:15:06.439011 16645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:06.443380 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 16563
W20260923 18:15:06.448725 16548 connection.cc:535] client connection to 127.0.0.1:38547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38547): Connection reset by peer (error 104)
2026-09-23T18:15:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:06 dist-test-slave-wsx5 krb5kdc[16663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:06 dist-test-slave-wsx5 krb5kdc[16663](info): set up 2 sockets
Sep 23 18:15:06 dist-test-slave-wsx5 krb5kdc[16663](info): commencing operation
krb5kdc: starting...
W20260923 18:15:07.407404 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:07 dist-test-slave-wsx5 krb5kdc[16663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:15:07.537235 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:07.651624 16676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:07.651911 16676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:07.651955 16676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:07.656083 16676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:07.656158 16676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:07.656177 16676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:07.656194 16676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:07.656207 16676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:07.661676 16676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:07.662818 16676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:07.663901 16676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:07.673410 16682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:07.673406 16684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:07.673410 16681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:07.673782 16676 server_base.cc:1034] running on GCE node
I20260923 18:15:07.674518 16676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:07.675712 16676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:07.676901 16676 hybrid_clock.cc:648] HybridClock initialized: now 1790187307676872 us; error 44 us; skew 500 ppm
Sep 23 18:15:07 dist-test-slave-wsx5 krb5kdc[16663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:07.679754 16676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:15:07.680714 16676 webserver.cc:460] Webserver started at http://127.0.0.1:45465/ using document root <none> and password file <none>
I20260923 18:15:07.681308 16676 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:07.681380 16676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:07.681547 16676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:07.683593 16676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "89bdc4fc9b55443ea9022172f6d02f41"
format_stamp: "Formatted at 2026-09-23 18:15:07 on dist-test-slave-wsx5"
server_key: "41986a92f3366485ac77dfb0241a429c"
server_key_iv: "005db32c767cbe75725d2656f93c5886"
server_key_version: "encryptionkey@0"
I20260923 18:15:07.684125 16676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "89bdc4fc9b55443ea9022172f6d02f41"
format_stamp: "Formatted at 2026-09-23 18:15:07 on dist-test-slave-wsx5"
server_key: "41986a92f3366485ac77dfb0241a429c"
server_key_iv: "005db32c767cbe75725d2656f93c5886"
server_key_version: "encryptionkey@0"
I20260923 18:15:07.687968 16676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260923 18:15:07.690577 16691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:07.691856 16676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 18:15:07.691967 16676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "89bdc4fc9b55443ea9022172f6d02f41"
format_stamp: "Formatted at 2026-09-23 18:15:07 on dist-test-slave-wsx5"
server_key: "41986a92f3366485ac77dfb0241a429c"
server_key_iv: "005db32c767cbe75725d2656f93c5886"
server_key_version: "encryptionkey@0"
I20260923 18:15:07.692063 16676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:07.697041 16676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:07.700762 16676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:07.700947 16676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:07.708431 16676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38185
I20260923 18:15:07.708464 16743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38185 every 8 connection(s)
I20260923 18:15:07.709741 16676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:07.712596 16744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:07.713709 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 16676
I20260923 18:15:07.713848 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:07.714150 27951 external_mini_cluster.cc:1428] Setting key 6bb240b8d91c4eaf865df59a0e3068b6
I20260923 18:15:07.719590 16744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41: Bootstrap starting.
I20260923 18:15:07.720616 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:07.715746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:39267 (local address 127.0.0.1:38185)
0923 18:15:07.716249 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:07.716265 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:07.716304 (+ 39us) server_negotiation.cc:407] Connection header received
0923 18:15:07.717034 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:07.717060 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:07.717504 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:07.717964 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:07.719268 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:07.719273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:07.719287 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:07.719299 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:07.719362 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:07.719386 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:07.719529 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:07.719603 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 18:15:07.719782 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":135,"threads_started":1}
I20260923 18:15:07.722114 16744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:07.722949 16744 log.cc:824] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:07.723649 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:07.725003 16744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41: No bootstrap required, opened a new log
I20260923 18:15:07.730082 16744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89bdc4fc9b55443ea9022172f6d02f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260923 18:15:07.730461 16744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:07.730551 16744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89bdc4fc9b55443ea9022172f6d02f41, State: Initialized, Role: FOLLOWER
I20260923 18:15:07.731107 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:07.729007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:48441 (local address 127.0.0.1:38185)
0923 18:15:07.729231 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:07.729235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:07.729251 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:15:07.729312 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:07.729315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:07.729359 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:07.729454 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:07.730544 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:07.730548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:07.730551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:07.730568 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:07.730604 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:07.730620 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:07.730692 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:07.730919 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 18:15:07.730973 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 18:15:07.731118 16744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89bdc4fc9b55443ea9022172f6d02f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260923 18:15:07.731474 16744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:07.731581 16744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:07.731725 16744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:07.732910 16744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89bdc4fc9b55443ea9022172f6d02f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260923 18:15:07.733335 16744 leader_election.cc:302] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89bdc4fc9b55443ea9022172f6d02f41; no voters:
I20260923 18:15:07.733696 16744 leader_election.cc:288] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:07.733904 16754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:07.734117 16754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [term 1 LEADER]: Becoming Leader. State: Replica: 89bdc4fc9b55443ea9022172f6d02f41, State: Running, Role: LEADER
I20260923 18:15:07.734449 16754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89bdc4fc9b55443ea9022172f6d02f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260923 18:15:07.735047 16744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:07.736301 16756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89bdc4fc9b55443ea9022172f6d02f41. Latest consensus state: current_term: 1 leader_uuid: "89bdc4fc9b55443ea9022172f6d02f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89bdc4fc9b55443ea9022172f6d02f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } } }
I20260923 18:15:07.736443 16756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:07.736897 16764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:07.736938 16755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89bdc4fc9b55443ea9022172f6d02f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89bdc4fc9b55443ea9022172f6d02f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } } }
I20260923 18:15:07.737041 16755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:07.740335 16764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:07.747116 16764 catalog_manager.cc:1347] Generated new cluster ID: 6f3e3a5d9baa46f8bc5dc49046dfb072
I20260923 18:15:07.747200 16764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:07.757264 16764 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:07.757638 16764 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:07.775862 16764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89bdc4fc9b55443ea9022172f6d02f41: Generated new TSK 0
I20260923 18:15:07.776657 16764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:07.779549 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 16676
2026-09-23T18:15:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:07 dist-test-slave-wsx5 krb5kdc[16775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:07 dist-test-slave-wsx5 krb5kdc[16775](info): set up 2 sockets
Sep 23 18:15:07 dist-test-slave-wsx5 krb5kdc[16775](info): commencing operation
krb5kdc: starting...
W20260923 18:15:08.739441 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:08 dist-test-slave-wsx5 krb5kdc[16775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:15:08.862238 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:08.974265 16788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:08.974555 16788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:08.974597 16788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:08.978751 16788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:08.978997 16788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:08.979069 16788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:08.979089 16788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:08.979110 16788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:08.984859 16788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:08.986025 16788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:08.987051 16788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:08.996676 16793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:08.996690 16794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:08.996703 16796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:08.997108 16788 server_base.cc:1034] running on GCE node
I20260923 18:15:08.997810 16788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:08.999058 16788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:09.000283 16788 hybrid_clock.cc:648] HybridClock initialized: now 1790187309000253 us; error 45 us; skew 500 ppm
Sep 23 18:15:08 dist-test-slave-wsx5 krb5kdc[16775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:09.003427 16788 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:15:09.004621 16788 webserver.cc:460] Webserver started at http://127.0.0.1:45611/ using document root <none> and password file <none>
I20260923 18:15:09.005249 16788 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:09.005329 16788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:09.005544 16788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:09.007687 16788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "91f7c08bd2f645f595631631009f835e"
format_stamp: "Formatted at 2026-09-23 18:15:09 on dist-test-slave-wsx5"
server_key: "b2f731f1bc30f43135ed08ed8c1ed6d4"
server_key_iv: "cce1b04717991ba5fea0a96eb31b5bae"
server_key_version: "encryptionkey@0"
I20260923 18:15:09.008263 16788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "91f7c08bd2f645f595631631009f835e"
format_stamp: "Formatted at 2026-09-23 18:15:09 on dist-test-slave-wsx5"
server_key: "b2f731f1bc30f43135ed08ed8c1ed6d4"
server_key_iv: "cce1b04717991ba5fea0a96eb31b5bae"
server_key_version: "encryptionkey@0"
I20260923 18:15:09.012367 16788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260923 18:15:09.015137 16804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:09.016502 16788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 18:15:09.016638 16788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "91f7c08bd2f645f595631631009f835e"
format_stamp: "Formatted at 2026-09-23 18:15:09 on dist-test-slave-wsx5"
server_key: "b2f731f1bc30f43135ed08ed8c1ed6d4"
server_key_iv: "cce1b04717991ba5fea0a96eb31b5bae"
server_key_version: "encryptionkey@0"
I20260923 18:15:09.016747 16788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:09.050722 16788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:09.054447 16788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:09.054663 16788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:09.061914 16788 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42239
I20260923 18:15:09.061942 16856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42239 every 8 connection(s)
I20260923 18:15:09.062916 16788 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:09.065883 16857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:09.069373 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 16788
I20260923 18:15:09.069530 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:09.069829 27951 external_mini_cluster.cc:1428] Setting key 98dd1bdb961ade1b1fc722c7a634fcfe
I20260923 18:15:09.072340 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e: Bootstrap starting.
I20260923 18:15:09.074693 16857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:09.075456 16857 log.cc:824] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:09.077567 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e: No bootstrap required, opened a new log
Sep 23 18:15:09 dist-test-slave-wsx5 krb5kdc[16775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 18:15:09.081326 16857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f7c08bd2f645f595631631009f835e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260923 18:15:09.081555 16857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:09.081590 16857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f7c08bd2f645f595631631009f835e, State: Initialized, Role: FOLLOWER
I20260923 18:15:09.082033 16857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f7c08bd2f645f595631631009f835e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260923 18:15:09.082170 16857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:09.082237 16857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:09.082340 16857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:09.083515 16857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f7c08bd2f645f595631631009f835e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260923 18:15:09.083901 16857 leader_election.cc:302] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91f7c08bd2f645f595631631009f835e; no voters:
I20260923 18:15:09.084266 16857 leader_election.cc:288] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:09.084372 16862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:09.084578 16862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [term 1 LEADER]: Becoming Leader. State: Replica: 91f7c08bd2f645f595631631009f835e, State: Running, Role: LEADER
I20260923 18:15:09.084900 16862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f7c08bd2f645f595631631009f835e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260923 18:15:09.085091 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:09.071443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:53769 (local address 127.0.0.1:42239)
0923 18:15:09.071968 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:09.071985 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:09.072024 (+ 39us) server_negotiation.cc:407] Connection header received
0923 18:15:09.072788 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:09.072810 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:09.073227 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:09.073599 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:09.074377 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:09.075495 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:09.076550 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:09.077014 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:09.077181 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:09.080000 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:09.080005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:09.080020 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:09.080052 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:09.081775 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:09.082285 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:09.082291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:09.082297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:09.082443 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:09.082834 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:09.082837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:09.082839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:09.083371 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:09.083593 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:09.083882 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 18:15:09.084117 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":178,"threads_started":1}
I20260923 18:15:09.085564 16857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:09.086098 16862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f7c08bd2f645f595631631009f835e. Latest consensus state: current_term: 1 leader_uuid: "91f7c08bd2f645f595631631009f835e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f7c08bd2f645f595631631009f835e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } } }
I20260923 18:15:09.086210 16862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:09.086530 16864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91f7c08bd2f645f595631631009f835e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f7c08bd2f645f595631631009f835e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } } }
I20260923 18:15:09.086612 16864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:09.088585 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 18:15:09.090621 16877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:15:09.090718 16877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:15:09.090785 16867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:09.091622 16867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:09.098044 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:09.090557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:47837 (local address 127.0.0.1:42239)
0923 18:15:09.090666 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:09.090670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:09.090910 (+ 240us) server_negotiation.cc:407] Connection header received
0923 18:15:09.091045 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:09.091048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:09.091090 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:09.091153 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:09.091828 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:09.092340 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:09.093169 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:09.093396 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:09.093483 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:09.094832 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:09.094835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:09.094837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:09.094861 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:09.096299 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:09.096775 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:09.096778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:09.096779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:09.096857 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:09.097184 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:09.097187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:09.097189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:09.097369 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:09.097497 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:09.097793 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 18:15:09.097886 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 18:15:09.098405 16867 catalog_manager.cc:1347] Generated new cluster ID: cf1a43d8b8ec4d68babbc258781421f8
I20260923 18:15:09.098475 16867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:09.117931 16867 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:09.119249 16867 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:09.126884 16867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91f7c08bd2f645f595631631009f835e: Generated new TSK 0
I20260923 18:15:09.127596 16867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:09.136600 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 16788
W20260923 18:15:09.141264 16773 connection.cc:535] client connection to 127.0.0.1:42239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42239): Connection reset by peer (error 104)
2026-09-23T18:15:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:09 dist-test-slave-wsx5 krb5kdc[16888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:09 dist-test-slave-wsx5 krb5kdc[16888](info): set up 2 sockets
Sep 23 18:15:09 dist-test-slave-wsx5 krb5kdc[16888](info): commencing operation
krb5kdc: starting...
W20260923 18:15:10.099319 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:10 dist-test-slave-wsx5 krb5kdc[16888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:15:10.227433 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:10.344308 16901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:10.344609 16901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:10.344655 16901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:10.348852 16901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:10.348937 16901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:10.348956 16901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:10.348971 16901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:10.348985 16901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:10.354513 16901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:10.355625 16901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:10.356681 16901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:15:10.365815 16901 server_base.cc:1034] running on GCE node
W20260923 18:15:10.365818 16909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:10.365839 16906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:10.365854 16907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:10.366644 16901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:10.367821 16901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:10.368995 16901 hybrid_clock.cc:648] HybridClock initialized: now 1790187310368976 us; error 33 us; skew 500 ppm
Sep 23 18:15:10 dist-test-slave-wsx5 krb5kdc[16888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:10.372004 16901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:15:10.373021 16901 webserver.cc:460] Webserver started at http://127.0.0.1:33609/ using document root <none> and password file <none>
I20260923 18:15:10.373598 16901 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:10.373660 16901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:10.373876 16901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:10.375991 16901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "5bf44445be204e4f8ba02292f8ba1b5b"
format_stamp: "Formatted at 2026-09-23 18:15:10 on dist-test-slave-wsx5"
server_key: "744da43df55b9f563339ba954c990fb1"
server_key_iv: "a107dc274f56853e6cefe74723c9d8ee"
server_key_version: "encryptionkey@0"
I20260923 18:15:10.376573 16901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "5bf44445be204e4f8ba02292f8ba1b5b"
format_stamp: "Formatted at 2026-09-23 18:15:10 on dist-test-slave-wsx5"
server_key: "744da43df55b9f563339ba954c990fb1"
server_key_iv: "a107dc274f56853e6cefe74723c9d8ee"
server_key_version: "encryptionkey@0"
I20260923 18:15:10.380566 16901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 18:15:10.383239 16916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:10.384619 16901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 18:15:10.384737 16901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "5bf44445be204e4f8ba02292f8ba1b5b"
format_stamp: "Formatted at 2026-09-23 18:15:10 on dist-test-slave-wsx5"
server_key: "744da43df55b9f563339ba954c990fb1"
server_key_iv: "a107dc274f56853e6cefe74723c9d8ee"
server_key_version: "encryptionkey@0"
I20260923 18:15:10.384859 16901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:10.398305 16901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:10.401734 16901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:10.401925 16901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:10.408926 16901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46177
I20260923 18:15:10.408944 16968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46177 every 8 connection(s)
I20260923 18:15:10.409955 16901 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:10.412811 16969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:10.414258 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 16901
I20260923 18:15:10.414387 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:10.414671 27951 external_mini_cluster.cc:1428] Setting key 5e678e17df71b57c191390bf66b3259b
I20260923 18:15:10.419000 16969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b: Bootstrap starting.
I20260923 18:15:10.421253 16969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:10.421954 16969 log.cc:824] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:10.423658 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:10.416139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:54379 (local address 127.0.0.1:46177)
0923 18:15:10.416608 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:10.416624 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:10.416659 (+ 35us) server_negotiation.cc:407] Connection header received
0923 18:15:10.417414 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:10.417436 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:10.417791 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:10.418113 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:10.418515 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:10.419498 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:10.420374 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:10.420855 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:10.421001 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:10.421889 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:10.421894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:10.421907 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:10.422336 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:10.422408 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:10.422439 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:10.422576 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:10.422767 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 18:15:10.422912 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":144,"threads_started":1}
I20260923 18:15:10.423954 16969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b: No bootstrap required, opened a new log
I20260923 18:15:10.426707 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:10.428231 16969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf44445be204e4f8ba02292f8ba1b5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260923 18:15:10.428540 16969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:10.428603 16969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf44445be204e4f8ba02292f8ba1b5b, State: Initialized, Role: FOLLOWER
I20260923 18:15:10.429104 16969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf44445be204e4f8ba02292f8ba1b5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260923 18:15:10.429282 16969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:10.429327 16969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:10.429417 16969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:10.430573 16969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf44445be204e4f8ba02292f8ba1b5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260923 18:15:10.430938 16969 leader_election.cc:302] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bf44445be204e4f8ba02292f8ba1b5b; no voters:
I20260923 18:15:10.431221 16969 leader_election.cc:288] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:10.431352 16979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:10.431560 16979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [term 1 LEADER]: Becoming Leader. State: Replica: 5bf44445be204e4f8ba02292f8ba1b5b, State: Running, Role: LEADER
I20260923 18:15:10.431957 16979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf44445be204e4f8ba02292f8ba1b5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260923 18:15:10.432449 16969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:10.433660 16981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bf44445be204e4f8ba02292f8ba1b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf44445be204e4f8ba02292f8ba1b5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } } }
I20260923 18:15:10.433806 16981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:10.434072 16981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bf44445be204e4f8ba02292f8ba1b5b. Latest consensus state: current_term: 1 leader_uuid: "5bf44445be204e4f8ba02292f8ba1b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf44445be204e4f8ba02292f8ba1b5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } } }
I20260923 18:15:10.434160 16981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:10.434226 16989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:10.434351 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:10.428935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:49141 (local address 127.0.0.1:46177)
0923 18:15:10.429047 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:10.429052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:10.429291 (+ 239us) server_negotiation.cc:407] Connection header received
0923 18:15:10.429485 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:10.429488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:10.429545 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:10.429629 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:10.429873 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:10.430612 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:10.431682 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:10.432024 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:10.432100 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:10.433357 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:10.433361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:10.433364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:10.433380 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:10.433412 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:10.433428 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:10.433489 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:10.433725 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 18:15:10.433766 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 18:15:10.437664 16989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:10.443019 16989 catalog_manager.cc:1347] Generated new cluster ID: 1fd3f66ba568475d8200564bfaeec5d4
I20260923 18:15:10.443112 16989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:10.448040 16989 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:10.449323 16989 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:10.459394 16989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bf44445be204e4f8ba02292f8ba1b5b: Generated new TSK 0
I20260923 18:15:10.460240 16989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:10.466284 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 16901
W20260923 18:15:10.471115 16886 connection.cc:535] client connection to 127.0.0.1:46177 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46177): Connection reset by peer (error 104)
2026-09-23T18:15:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:10 dist-test-slave-wsx5 krb5kdc[17000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:10 dist-test-slave-wsx5 krb5kdc[17000](info): set up 2 sockets
Sep 23 18:15:10 dist-test-slave-wsx5 krb5kdc[17000](info): commencing operation
krb5kdc: starting...
W20260923 18:15:11.443264 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:11 dist-test-slave-wsx5 krb5kdc[17000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 18:15:11.565274 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:11.681005 17014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:11.681348 17014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:11.681399 17014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:11.685621 17014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:11.685693 17014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:11.685712 17014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:11.685736 17014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:11.685751 17014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:11.691335 17014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37391
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:11.692462 17014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:11.693857 17014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:11.703091 17019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:11.703091 17022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:11.703091 17020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:11.703280 17014 server_base.cc:1034] running on GCE node
I20260923 18:15:11.703951 17014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:11.705471 17014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:11.706686 17014 hybrid_clock.cc:648] HybridClock initialized: now 1790187311706663 us; error 35 us; skew 500 ppm
Sep 23 18:15:11 dist-test-slave-wsx5 krb5kdc[17000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:11.713864 17014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 18:15:11.715092 17014 webserver.cc:460] Webserver started at http://127.0.0.1:33885/ using document root <none> and password file <none>
I20260923 18:15:11.715695 17014 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:11.715788 17014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:11.716012 17014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:11.718253 17014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "cb4a275d46184144b61b4fa9eacb2a36"
format_stamp: "Formatted at 2026-09-23 18:15:11 on dist-test-slave-wsx5"
server_key: "d3b1668af63bad321386cf02e2b143a6"
server_key_iv: "48d5789bd5bdc78800894bcb48e3f287"
server_key_version: "encryptionkey@0"
I20260923 18:15:11.718832 17014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "cb4a275d46184144b61b4fa9eacb2a36"
format_stamp: "Formatted at 2026-09-23 18:15:11 on dist-test-slave-wsx5"
server_key: "d3b1668af63bad321386cf02e2b143a6"
server_key_iv: "48d5789bd5bdc78800894bcb48e3f287"
server_key_version: "encryptionkey@0"
I20260923 18:15:11.723129 17014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 18:15:11.726016 17029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:11.727772 17014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 18:15:11.727996 17014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "cb4a275d46184144b61b4fa9eacb2a36"
format_stamp: "Formatted at 2026-09-23 18:15:11 on dist-test-slave-wsx5"
server_key: "d3b1668af63bad321386cf02e2b143a6"
server_key_iv: "48d5789bd5bdc78800894bcb48e3f287"
server_key_version: "encryptionkey@0"
I20260923 18:15:11.728209 17014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:11.735772 17014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:11.739912 17014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:11.740115 17014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:11.746884 17014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37391
I20260923 18:15:11.746944 17081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37391 every 8 connection(s)
I20260923 18:15:11.747859 17014 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:11.750759 17082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:11.751995 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 17014
I20260923 18:15:11.752115 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:11.752365 27951 external_mini_cluster.cc:1428] Setting key f99b4ca0dc11871839ace528c89b698c
I20260923 18:15:11.756943 17082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36: Bootstrap starting.
I20260923 18:15:11.757640 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:11.753879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:34139 (local address 127.0.0.1:37391)
0923 18:15:11.754318 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:11.754335 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:11.754370 (+ 35us) server_negotiation.cc:407] Connection header received
0923 18:15:11.755003 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:11.755020 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:11.755311 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:11.755620 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:11.756376 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:11.756381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:11.756394 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:11.756404 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:11.756458 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:11.756477 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:11.756601 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:11.756676 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 18:15:11.756836 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":126,"threads_started":1}
I20260923 18:15:11.759462 17082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:11.760257 17082 log.cc:824] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:11.760576 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:11.762358 17082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36: No bootstrap required, opened a new log
I20260923 18:15:11.764271 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:11.762559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:54111 (local address 127.0.0.1:37391)
0923 18:15:11.762668 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:11.762672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:11.762863 (+ 191us) server_negotiation.cc:407] Connection header received
0923 18:15:11.763038 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:11.763041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:11.763078 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:11.763142 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:11.763865 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:11.763868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:11.763870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:11.763882 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:11.763908 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:11.763918 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:11.763993 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:11.764079 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 18:15:11.764122 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 18:15:11.766368 17082 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4a275d46184144b61b4fa9eacb2a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } }
I20260923 18:15:11.766610 17082 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:11.766651 17082 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4a275d46184144b61b4fa9eacb2a36, State: Initialized, Role: FOLLOWER
I20260923 18:15:11.767028 17082 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4a275d46184144b61b4fa9eacb2a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } }
I20260923 18:15:11.767139 17082 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:11.767184 17082 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:11.767264 17082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:11.768167 17082 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4a275d46184144b61b4fa9eacb2a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } }
I20260923 18:15:11.768496 17082 leader_election.cc:302] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb4a275d46184144b61b4fa9eacb2a36; no voters:
I20260923 18:15:11.768762 17082 leader_election.cc:288] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:11.768879 17092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:11.769099 17092 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [term 1 LEADER]: Becoming Leader. State: Replica: cb4a275d46184144b61b4fa9eacb2a36, State: Running, Role: LEADER
I20260923 18:15:11.769445 17092 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4a275d46184144b61b4fa9eacb2a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } }
I20260923 18:15:11.770004 17082 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:11.771400 17094 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb4a275d46184144b61b4fa9eacb2a36. Latest consensus state: current_term: 1 leader_uuid: "cb4a275d46184144b61b4fa9eacb2a36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4a275d46184144b61b4fa9eacb2a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } } }
I20260923 18:15:11.771541 17094 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:11.771849 17093 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb4a275d46184144b61b4fa9eacb2a36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4a275d46184144b61b4fa9eacb2a36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37391 } } }
I20260923 18:15:11.771944 17093 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36 [sys.catalog]: This master's current role is: LEADER
W20260923 18:15:11.774466 17107 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:15:11.774541 17107 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:15:11.774621 17105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:11.775331 17105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:11.781544 17105 catalog_manager.cc:1347] Generated new cluster ID: 9bebb6bb39f6408d83b9d8650b86a2a7
I20260923 18:15:11.781631 17105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:11.793870 17105 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:11.794200 17105 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:11.798985 17105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb4a275d46184144b61b4fa9eacb2a36: Generated new TSK 0
I20260923 18:15:11.799736 17105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:11.803345 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 17014
2026-09-23T18:15:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:11 dist-test-slave-wsx5 krb5kdc[17113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:11 dist-test-slave-wsx5 krb5kdc[17113](info): set up 2 sockets
Sep 23 18:15:11 dist-test-slave-wsx5 krb5kdc[17113](info): commencing operation
krb5kdc: starting...
W20260923 18:15:12.763441 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:12 dist-test-slave-wsx5 krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:15:12.886519 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:37469
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:12.999617 17126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:12.999915 17126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:12.999966 17126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:13.004241 17126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:13.004319 17126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:13.004338 17126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:13.004362 17126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:13.004382 17126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:13.010038 17126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:37469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:37469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:13.011145 17126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:13.012157 17126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:13.020920 17131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:13.020931 17132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:13.020925 17134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:13.021241 17126 server_base.cc:1034] running on GCE node
I20260923 18:15:13.021929 17126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:13.023102 17126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:13.024283 17126 hybrid_clock.cc:648] HybridClock initialized: now 1790187313024257 us; error 45 us; skew 500 ppm
Sep 23 18:15:13 dist-test-slave-wsx5 krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187313, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:13.027279 17126 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:15:13.028247 17126 webserver.cc:460] Webserver started at http://127.27.75.254:41121/ using document root <none> and password file <none>
I20260923 18:15:13.028805 17126 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:13.028878 17126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:13.029083 17126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:13.031183 17126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "98705b1cac824d7ab39d7069c28eb170"
format_stamp: "Formatted at 2026-09-23 18:15:13 on dist-test-slave-wsx5"
server_key: "204b85fd0b857f29e96cc4f36fc79f56"
server_key_iv: "86fcdba16ae471f19780737aa144036b"
server_key_version: "encryptionkey@0"
I20260923 18:15:13.031765 17126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "98705b1cac824d7ab39d7069c28eb170"
format_stamp: "Formatted at 2026-09-23 18:15:13 on dist-test-slave-wsx5"
server_key: "204b85fd0b857f29e96cc4f36fc79f56"
server_key_iv: "86fcdba16ae471f19780737aa144036b"
server_key_version: "encryptionkey@0"
I20260923 18:15:13.035730 17126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:15:13.038726 17141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:13.040230 17126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:15:13.040397 17126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "98705b1cac824d7ab39d7069c28eb170"
format_stamp: "Formatted at 2026-09-23 18:15:13 on dist-test-slave-wsx5"
server_key: "204b85fd0b857f29e96cc4f36fc79f56"
server_key_iv: "86fcdba16ae471f19780737aa144036b"
server_key_version: "encryptionkey@0"
I20260923 18:15:13.040521 17126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:13.053576 17126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:13.056874 17126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:13.057073 17126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:13.064137 17126 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:37469
I20260923 18:15:13.064170 17194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:37469 every 8 connection(s)
I20260923 18:15:13.065191 17126 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:13.068116 17195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:13.072868 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 17126
I20260923 18:15:13.073019 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:13.073354 27951 external_mini_cluster.cc:1428] Setting key 0a61afd721af5503c346eed945edb57c
I20260923 18:15:13.074906 17195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170: Bootstrap starting.
I20260923 18:15:13.077381 17195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:13.078249 17195 log.cc:824] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:13.080212 17195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170: No bootstrap required, opened a new log
Sep 23 18:15:13 dist-test-slave-wsx5 krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:15:13.084625 17195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98705b1cac824d7ab39d7069c28eb170" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37469 } }
I20260923 18:15:13.084954 17195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:13.085026 17195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98705b1cac824d7ab39d7069c28eb170, State: Initialized, Role: FOLLOWER
I20260923 18:15:13.085601 17195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98705b1cac824d7ab39d7069c28eb170" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37469 } }
I20260923 18:15:13.085742 17195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:13.085817 17195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:13.085917 17195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:13.087101 17195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98705b1cac824d7ab39d7069c28eb170" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37469 } }
I20260923 18:15:13.087487 17195 leader_election.cc:302] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98705b1cac824d7ab39d7069c28eb170; no voters:
I20260923 18:15:13.087869 17195 leader_election.cc:288] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:13.087988 17200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:13.088243 17200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [term 1 LEADER]: Becoming Leader. State: Replica: 98705b1cac824d7ab39d7069c28eb170, State: Running, Role: LEADER
I20260923 18:15:13.088649 17200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98705b1cac824d7ab39d7069c28eb170" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37469 } }
I20260923 18:15:13.088887 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:13.074813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42573 (local address 127.27.75.254:37469)
0923 18:15:13.075257 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:13.075274 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:13.075311 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:15:13.076049 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:13.076068 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:13.076462 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:13.076767 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:13.077735 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:13.078796 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:13.079706 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:13.080208 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:13.080356 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:13.083175 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:13.083184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:13.083206 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:13.083255 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:13.085353 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:13.085975 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:13.085981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:13.085988 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:13.086134 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:13.086485 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:13.086488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:13.086490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:13.087046 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:13.087274 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:13.087648 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 18:15:13.087897 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260923 18:15:13.089252 17195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:13.090287 17202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98705b1cac824d7ab39d7069c28eb170. Latest consensus state: current_term: 1 leader_uuid: "98705b1cac824d7ab39d7069c28eb170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98705b1cac824d7ab39d7069c28eb170" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37469 } } }
I20260923 18:15:13.090418 17202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:13.090813 17209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:13.091477 17201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98705b1cac824d7ab39d7069c28eb170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98705b1cac824d7ab39d7069c28eb170" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37469 } } }
I20260923 18:15:13.091575 17201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:13.092445 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:13.094167 17209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:13.102334 17209 catalog_manager.cc:1347] Generated new cluster ID: b9166a88609240b9915cdbcbf34db5e8
I20260923 18:15:13.102454 17209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:13.103454 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:13.094925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54861 (local address 127.27.75.254:37469)
0923 18:15:13.095035 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:13.095040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:13.095225 (+ 185us) server_negotiation.cc:407] Connection header received
0923 18:15:13.095419 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:13.095422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:13.095467 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:13.095528 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:13.096186 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:13.096702 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:13.097714 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:13.098051 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:13.098125 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:13.100006 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:13.100009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:13.100011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:13.100037 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:13.101678 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:13.102223 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:13.102226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:13.102227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:13.102307 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:13.102636 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:13.102639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:13.102641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:13.102825 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:13.102953 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:13.103219 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 18:15:13.103310 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 18:15:13.140415 17209 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:13.141476 17209 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:13.145591 17209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98705b1cac824d7ab39d7069c28eb170: Generated new TSK 0
I20260923 18:15:13.146279 17209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:13.152413 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 17126
W20260923 18:15:13.157037 17111 connection.cc:535] client connection to 127.27.75.254:37469 recv error: Network error: failed to read from TLS socket (remote: 127.27.75.254:37469): Connection reset by peer (error 104)
2026-09-23T18:15:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:13 dist-test-slave-wsx5 krb5kdc[17226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:13 dist-test-slave-wsx5 krb5kdc[17226](info): set up 2 sockets
Sep 23 18:15:13 dist-test-slave-wsx5 krb5kdc[17226](info): commencing operation
krb5kdc: starting...
W20260923 18:15:14.115428 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:14 dist-test-slave-wsx5 krb5kdc[17226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:15:14.240015 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:35863
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:35863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:14.354375 17239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:14.354683 17239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:14.354753 17239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:14.358994 17239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:14.359081 17239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:14.359108 17239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:14.359129 17239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:14.359148 17239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:14.364768 17239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:35863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:35863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:14.365928 17239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:14.366951 17239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:14.375507 17248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:14.375494 17245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:14.375527 17246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:14.375689 17239 server_base.cc:1034] running on GCE node
I20260923 18:15:14.376469 17239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:14.377740 17239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:14.378921 17239 hybrid_clock.cc:648] HybridClock initialized: now 1790187314378907 us; error 29 us; skew 500 ppm
Sep 23 18:15:14 dist-test-slave-wsx5 krb5kdc[17226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:14.381690 17239 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:15:14.382644 17239 webserver.cc:460] Webserver started at http://127.27.75.254:40113/ using document root <none> and password file <none>
I20260923 18:15:14.383179 17239 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:14.383241 17239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:14.383440 17239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:14.385565 17239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "8e68429343d849c69755d3c530ed4bac"
format_stamp: "Formatted at 2026-09-23 18:15:14 on dist-test-slave-wsx5"
server_key: "4da4c224b71581782bfa4b84b6d9e9ab"
server_key_iv: "c2b625b938d196cb4b774cf2c6bfd53c"
server_key_version: "encryptionkey@0"
I20260923 18:15:14.386164 17239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "8e68429343d849c69755d3c530ed4bac"
format_stamp: "Formatted at 2026-09-23 18:15:14 on dist-test-slave-wsx5"
server_key: "4da4c224b71581782bfa4b84b6d9e9ab"
server_key_iv: "c2b625b938d196cb4b774cf2c6bfd53c"
server_key_version: "encryptionkey@0"
I20260923 18:15:14.390286 17239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:15:14.393033 17255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:14.394402 17239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 18:15:14.394516 17239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "8e68429343d849c69755d3c530ed4bac"
format_stamp: "Formatted at 2026-09-23 18:15:14 on dist-test-slave-wsx5"
server_key: "4da4c224b71581782bfa4b84b6d9e9ab"
server_key_iv: "c2b625b938d196cb4b774cf2c6bfd53c"
server_key_version: "encryptionkey@0"
I20260923 18:15:14.394629 17239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:14.418107 17239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:14.421609 17239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:14.421809 17239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:14.429196 17239 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:35863
I20260923 18:15:14.429226 17307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:35863 every 8 connection(s)
I20260923 18:15:14.430213 17239 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:14.433175 17308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:14.436700 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 17239
I20260923 18:15:14.436874 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:14.437198 27951 external_mini_cluster.cc:1428] Setting key 678ee80e9d3fab5201d061ae9cf3c381
I20260923 18:15:14.439623 17308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac: Bootstrap starting.
I20260923 18:15:14.442118 17308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:14.442919 17308 log.cc:824] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:14.445309 17308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac: No bootstrap required, opened a new log
I20260923 18:15:14.447546 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:14.438883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56505 (local address 127.27.75.254:35863)
0923 18:15:14.439403 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:14.439423 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:14.439461 (+ 38us) server_negotiation.cc:407] Connection header received
0923 18:15:14.440200 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:14.440220 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:14.440603 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:14.441078 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:14.441573 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:14.442675 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:14.443673 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:14.444170 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:14.444357 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:14.445400 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:14.445469 (+ 69us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:14.445496 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:14.445945 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:14.446008 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:14.446034 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:14.446177 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:14.446379 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 18:15:14.446528 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":155,"threads_started":1}
I20260923 18:15:14.448534 17308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e68429343d849c69755d3c530ed4bac" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35863 } }
I20260923 18:15:14.448781 17308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:14.448840 17308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e68429343d849c69755d3c530ed4bac, State: Initialized, Role: FOLLOWER
I20260923 18:15:14.449290 17308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e68429343d849c69755d3c530ed4bac" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35863 } }
I20260923 18:15:14.449427 17308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:14.449537 17308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:14.449649 17308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:14.450687 17308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e68429343d849c69755d3c530ed4bac" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35863 } }
I20260923 18:15:14.450773 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:14.451045 17308 leader_election.cc:302] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e68429343d849c69755d3c530ed4bac; no voters:
I20260923 18:15:14.451359 17308 leader_election.cc:288] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:14.451440 17313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:14.451606 17313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [term 1 LEADER]: Becoming Leader. State: Replica: 8e68429343d849c69755d3c530ed4bac, State: Running, Role: LEADER
I20260923 18:15:14.451967 17313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e68429343d849c69755d3c530ed4bac" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35863 } }
I20260923 18:15:14.452715 17308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:14.453663 17314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e68429343d849c69755d3c530ed4bac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e68429343d849c69755d3c530ed4bac" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35863 } } }
I20260923 18:15:14.453794 17314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:14.454097 17315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e68429343d849c69755d3c530ed4bac. Latest consensus state: current_term: 1 leader_uuid: "8e68429343d849c69755d3c530ed4bac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e68429343d849c69755d3c530ed4bac" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 35863 } } }
I20260923 18:15:14.454175 17315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:14.454164 17327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:14.457659 17327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:14.459937 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:14.452879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53505 (local address 127.27.75.254:35863)
0923 18:15:14.452992 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:14.452997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:14.455020 (+ 2023us) server_negotiation.cc:407] Connection header received
0923 18:15:14.455221 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:14.455225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:14.455291 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:14.455386 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:14.455716 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:14.456497 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:14.457998 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:14.458325 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:14.458434 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:14.459433 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:14.459437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:14.459439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:14.459456 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:14.459489 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:14.459503 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:14.459586 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:14.459752 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 18:15:14.459800 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 18:15:14.464191 17327 catalog_manager.cc:1347] Generated new cluster ID: e6fad85b04264e719d478116e8987d02
I20260923 18:15:14.464300 17327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:14.477818 17327 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:14.479128 17327 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:14.486173 17327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e68429343d849c69755d3c530ed4bac: Generated new TSK 0
I20260923 18:15:14.487087 17327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:14.491415 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 17239
W20260923 18:15:14.496323 17224 connection.cc:535] client connection to 127.27.75.254:35863 recv error: Network error: failed to read from TLS socket (remote: 127.27.75.254:35863): Connection reset by peer (error 104)
2026-09-23T18:15:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:14 dist-test-slave-wsx5 krb5kdc[17339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:14 dist-test-slave-wsx5 krb5kdc[17339](info): set up 2 sockets
Sep 23 18:15:14 dist-test-slave-wsx5 krb5kdc[17339](info): commencing operation
krb5kdc: starting...
W20260923 18:15:15.459384 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:15 dist-test-slave-wsx5 krb5kdc[17339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:15:15.582458 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34647
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:34647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:15.693989 17352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:15.694299 17352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:15.694342 17352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:15.698562 17352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:15.698637 17352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:15.698654 17352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:15.698670 17352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:15.698684 17352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:15.704265 17352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:34647
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:15.705500 17352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:15.706513 17352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:15.715663 17357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:15.715664 17358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:15.715669 17360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:15.716063 17352 server_base.cc:1034] running on GCE node
I20260923 18:15:15.716696 17352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:15.717981 17352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:15.719231 17352 hybrid_clock.cc:648] HybridClock initialized: now 1790187315719205 us; error 41 us; skew 500 ppm
Sep 23 18:15:15 dist-test-slave-wsx5 krb5kdc[17339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:15.722414 17352 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:15:15.723448 17352 webserver.cc:460] Webserver started at http://127.27.75.254:36771/ using document root <none> and password file <none>
I20260923 18:15:15.723997 17352 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:15.724062 17352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:15.724263 17352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:15.726624 17352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "c0f111bf8da34568954579338fd13b9c"
format_stamp: "Formatted at 2026-09-23 18:15:15 on dist-test-slave-wsx5"
server_key: "dd8511eff11e89a9504ca0fc18f12920"
server_key_iv: "198b4d4f3b18411eaee4261ec9d5ed28"
server_key_version: "encryptionkey@0"
I20260923 18:15:15.727309 17352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "c0f111bf8da34568954579338fd13b9c"
format_stamp: "Formatted at 2026-09-23 18:15:15 on dist-test-slave-wsx5"
server_key: "dd8511eff11e89a9504ca0fc18f12920"
server_key_iv: "198b4d4f3b18411eaee4261ec9d5ed28"
server_key_version: "encryptionkey@0"
I20260923 18:15:15.732255 17352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 18:15:15.734925 17367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:15.736209 17352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260923 18:15:15.736334 17352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "c0f111bf8da34568954579338fd13b9c"
format_stamp: "Formatted at 2026-09-23 18:15:15 on dist-test-slave-wsx5"
server_key: "dd8511eff11e89a9504ca0fc18f12920"
server_key_iv: "198b4d4f3b18411eaee4261ec9d5ed28"
server_key_version: "encryptionkey@0"
I20260923 18:15:15.736443 17352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:15.741112 17352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:15.744330 17352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:15.744517 17352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:15.751550 17352 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:34647
I20260923 18:15:15.751578 17421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:34647 every 8 connection(s)
I20260923 18:15:15.752517 17352 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:15.755501 17422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:15.759037 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 17352
I20260923 18:15:15.759186 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:15.759469 27951 external_mini_cluster.cc:1428] Setting key f7af3bc5db34a3837a668ad632db030a
I20260923 18:15:15.761726 17422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c: Bootstrap starting.
I20260923 18:15:15.764266 17422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:15.765280 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:15.761009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60121 (local address 127.27.75.254:34647)
0923 18:15:15.761511 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:15.761527 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:15.761563 (+ 36us) server_negotiation.cc:407] Connection header received
0923 18:15:15.762253 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:15.762272 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:15.762568 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:15.762865 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:15.763746 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:15.763752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:15.763765 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:15.763778 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:15.763916 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:15.763961 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:15.764060 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:15.764245 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 18:15:15.764427 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":132,"threads_started":1}
I20260923 18:15:15.765295 17422 log.cc:824] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:15.767529 17422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c: No bootstrap required, opened a new log
I20260923 18:15:15.768353 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:15.772423 17422 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f111bf8da34568954579338fd13b9c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34647 } }
I20260923 18:15:15.772943 17422 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:15.773032 17422 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f111bf8da34568954579338fd13b9c, State: Initialized, Role: FOLLOWER
I20260923 18:15:15.773176 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:15.770899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43255 (local address 127.27.75.254:34647)
0923 18:15:15.771044 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:15.771050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:15.771363 (+ 313us) server_negotiation.cc:407] Connection header received
0923 18:15:15.771470 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:15.771473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:15.771509 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:15.771578 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:15.772345 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:15.772349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:15.772352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:15.772365 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:15.772395 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:15.772406 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:15.772614 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:15.772852 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 18:15:15.772908 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 18:15:15.773612 17422 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f111bf8da34568954579338fd13b9c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34647 } }
I20260923 18:15:15.773762 17422 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:15.773833 17422 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:15.773936 17422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:15.774911 17422 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f111bf8da34568954579338fd13b9c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34647 } }
I20260923 18:15:15.775271 17422 leader_election.cc:302] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0f111bf8da34568954579338fd13b9c; no voters:
I20260923 18:15:15.775576 17422 leader_election.cc:288] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:15.775686 17432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:15.775913 17432 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [term 1 LEADER]: Becoming Leader. State: Replica: c0f111bf8da34568954579338fd13b9c, State: Running, Role: LEADER
I20260923 18:15:15.776362 17432 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f111bf8da34568954579338fd13b9c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34647 } }
I20260923 18:15:15.776880 17422 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:15.778041 17434 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0f111bf8da34568954579338fd13b9c. Latest consensus state: current_term: 1 leader_uuid: "c0f111bf8da34568954579338fd13b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f111bf8da34568954579338fd13b9c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34647 } } }
I20260923 18:15:15.778177 17434 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:15.778591 17441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:15.778566 17433 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0f111bf8da34568954579338fd13b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f111bf8da34568954579338fd13b9c" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34647 } } }
I20260923 18:15:15.778697 17433 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:15.781656 17441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:15.787842 17441 catalog_manager.cc:1347] Generated new cluster ID: 0e4fac9d87ca4f2282e36a31ae4e27e7
I20260923 18:15:15.787930 17441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:15.800379 17441 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:15.800733 17441 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:15.809229 17441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0f111bf8da34568954579338fd13b9c: Generated new TSK 0
I20260923 18:15:15.810065 17441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:15.818742 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 17352
2026-09-23T18:15:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:15 dist-test-slave-wsx5 krb5kdc[17453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:15 dist-test-slave-wsx5 krb5kdc[17453](info): set up 2 sockets
Sep 23 18:15:15 dist-test-slave-wsx5 krb5kdc[17453](info): commencing operation
krb5kdc: starting...
W20260923 18:15:16.783468 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:16 dist-test-slave-wsx5 krb5kdc[17453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:15:16.903497 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:41319
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:41319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:17.014948 17466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:17.015244 17466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:17.015303 17466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:17.019488 17466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:17.019559 17466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:17.019586 17466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:17.019608 17466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:17.019634 17466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:17.025206 17466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:41319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:41319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:17.026345 17466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:17.027388 17466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:17.036437 17474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:17.036474 17471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:17.036521 17466 server_base.cc:1034] running on GCE node
W20260923 18:15:17.036474 17472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:17.037298 17466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:17.038576 17466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:17.039763 17466 hybrid_clock.cc:648] HybridClock initialized: now 1790187317039742 us; error 37 us; skew 500 ppm
Sep 23 18:15:17 dist-test-slave-wsx5 krb5kdc[17453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187317, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:17.046689 17466 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:15:17.047690 17466 webserver.cc:460] Webserver started at http://127.27.75.254:34081/ using document root <none> and password file <none>
I20260923 18:15:17.048269 17466 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:17.048336 17466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:17.048533 17466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:17.050722 17466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "2aa2c4f2342d41f698755120b644afe6"
format_stamp: "Formatted at 2026-09-23 18:15:17 on dist-test-slave-wsx5"
server_key: "d095427272cc28bbaab3c1526507273b"
server_key_iv: "22b9885ee1a421d626a5b5d140aa07b5"
server_key_version: "encryptionkey@0"
I20260923 18:15:17.051283 17466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "2aa2c4f2342d41f698755120b644afe6"
format_stamp: "Formatted at 2026-09-23 18:15:17 on dist-test-slave-wsx5"
server_key: "d095427272cc28bbaab3c1526507273b"
server_key_iv: "22b9885ee1a421d626a5b5d140aa07b5"
server_key_version: "encryptionkey@0"
I20260923 18:15:17.055227 17466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:15:17.057782 17481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:17.059106 17466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:15:17.059240 17466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "2aa2c4f2342d41f698755120b644afe6"
format_stamp: "Formatted at 2026-09-23 18:15:17 on dist-test-slave-wsx5"
server_key: "d095427272cc28bbaab3c1526507273b"
server_key_iv: "22b9885ee1a421d626a5b5d140aa07b5"
server_key_version: "encryptionkey@0"
I20260923 18:15:17.059347 17466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:17.087649 17466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:17.091069 17466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:17.091259 17466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:17.098141 17466 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:41319
I20260923 18:15:17.098242 17533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:41319 every 8 connection(s)
I20260923 18:15:17.099210 17466 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:17.100342 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 17466
I20260923 18:15:17.100492 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:17.100800 27951 external_mini_cluster.cc:1428] Setting key fabf685858e602918099eb784f2d0d11
I20260923 18:15:17.102540 17534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:17.109479 17534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6: Bootstrap starting.
Sep 23 18:15:17 dist-test-slave-wsx5 krb5kdc[17453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:15:17.111941 17534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:17.112879 17534 log.cc:824] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:17.115439 17534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6: No bootstrap required, opened a new log
I20260923 18:15:17.116617 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:17.102364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33419 (local address 127.27.75.254:41319)
0923 18:15:17.102816 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:17.102834 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:17.102877 (+ 43us) server_negotiation.cc:407] Connection header received
0923 18:15:17.103746 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:17.103768 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:17.104202 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:17.104586 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:17.105268 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:17.106777 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:17.107695 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:17.108435 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:17.108607 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:17.111186 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:17.111192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:17.111209 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:17.111245 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:17.113266 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:17.113823 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:17.113830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:17.113837 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:17.113995 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:17.114283 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:17.114286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:17.114287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:17.114818 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:17.115003 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:17.115290 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 18:15:17.115590 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":123,"threads_started":1}
I20260923 18:15:17.119181 17534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa2c4f2342d41f698755120b644afe6" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41319 } }
I20260923 18:15:17.119432 17534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:17.119494 17534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa2c4f2342d41f698755120b644afe6, State: Initialized, Role: FOLLOWER
I20260923 18:15:17.120036 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:17.119980 17534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa2c4f2342d41f698755120b644afe6" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41319 } }
I20260923 18:15:17.120122 17534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:17.120186 17534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:17.120289 17534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:17.121251 17534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa2c4f2342d41f698755120b644afe6" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41319 } }
I20260923 18:15:17.121596 17534 leader_election.cc:302] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aa2c4f2342d41f698755120b644afe6; no voters:
I20260923 18:15:17.121896 17534 leader_election.cc:288] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:17.121984 17543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:17.122185 17543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa2c4f2342d41f698755120b644afe6, State: Running, Role: LEADER
I20260923 18:15:17.122555 17543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa2c4f2342d41f698755120b644afe6" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41319 } }
I20260923 18:15:17.123320 17534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:17.123831 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aa2c4f2342d41f698755120b644afe6. Latest consensus state: current_term: 1 leader_uuid: "2aa2c4f2342d41f698755120b644afe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa2c4f2342d41f698755120b644afe6" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41319 } } }
I20260923 18:15:17.124003 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:17.124236 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aa2c4f2342d41f698755120b644afe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa2c4f2342d41f698755120b644afe6" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41319 } } }
I20260923 18:15:17.124375 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:17.124677 17551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:17.128182 17551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:17.131074 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:17.122092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53025 (local address 127.27.75.254:41319)
0923 18:15:17.122205 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:17.122209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:17.122426 (+ 217us) server_negotiation.cc:407] Connection header received
0923 18:15:17.122537 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:17.122540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:17.122587 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:17.122652 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:17.123301 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:17.124008 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:17.125206 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:17.125549 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:17.125656 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:17.127306 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:17.127310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:17.127313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:17.127342 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:17.129236 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:17.129744 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:17.129748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:17.129750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:17.129850 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:17.130154 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:17.130158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:17.130160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:17.130413 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:17.130544 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:17.130789 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 18:15:17.130921 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 18:15:17.134559 17551 catalog_manager.cc:1347] Generated new cluster ID: 01b27bfe35524b92b7c4fcfb929f2272
I20260923 18:15:17.134639 17551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:17.142118 17551 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:17.143373 17551 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:17.150115 17551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aa2c4f2342d41f698755120b644afe6: Generated new TSK 0
I20260923 18:15:17.150978 17551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:17.155617 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 17466
2026-09-23T18:15:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:17 dist-test-slave-wsx5 krb5kdc[17565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:17 dist-test-slave-wsx5 krb5kdc[17565](info): set up 2 sockets
Sep 23 18:15:17 dist-test-slave-wsx5 krb5kdc[17565](info): commencing operation
krb5kdc: starting...
W20260923 18:15:18.111373 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:18 dist-test-slave-wsx5 krb5kdc[17565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:15:18.236374 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34633
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:34347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:34633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:18.350068 17578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:18.350368 17578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:18.350411 17578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:18.354537 17578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:18.354615 17578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:18.354640 17578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:18.354655 17578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:18.354669 17578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:18.360137 17578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:34347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:34633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:34633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:18.361238 17578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:18.362236 17578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:18.371445 17583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:18.371536 17578 server_base.cc:1034] running on GCE node
W20260923 18:15:18.371420 17586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:18.371518 17584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:18.372354 17578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:18.373622 17578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:18.374859 17578 hybrid_clock.cc:648] HybridClock initialized: now 1790187318374770 us; error 57 us; skew 500 ppm
Sep 23 18:15:18 dist-test-slave-wsx5 krb5kdc[17565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187318, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:18.378010 17578 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:15:18.379041 17578 webserver.cc:460] Webserver started at http://127.27.75.254:33881/ using document root <none> and password file <none>
I20260923 18:15:18.379622 17578 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:18.379694 17578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:18.379895 17578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:18.382091 17578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "25cf96e5e7d84d5ab10340baf196ceb5"
format_stamp: "Formatted at 2026-09-23 18:15:18 on dist-test-slave-wsx5"
server_key: "12e4c0a1e7cfd9988e183d583e396565"
server_key_iv: "1c65178f0d17a9b5d6f604e6157bcc91"
server_key_version: "encryptionkey@0"
I20260923 18:15:18.382664 17578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "25cf96e5e7d84d5ab10340baf196ceb5"
format_stamp: "Formatted at 2026-09-23 18:15:18 on dist-test-slave-wsx5"
server_key: "12e4c0a1e7cfd9988e183d583e396565"
server_key_iv: "1c65178f0d17a9b5d6f604e6157bcc91"
server_key_version: "encryptionkey@0"
I20260923 18:15:18.386684 17578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:15:18.389492 17593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:18.390887 17578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 18:15:18.391021 17578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "25cf96e5e7d84d5ab10340baf196ceb5"
format_stamp: "Formatted at 2026-09-23 18:15:18 on dist-test-slave-wsx5"
server_key: "12e4c0a1e7cfd9988e183d583e396565"
server_key_iv: "1c65178f0d17a9b5d6f604e6157bcc91"
server_key_version: "encryptionkey@0"
I20260923 18:15:18.391135 17578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:18.414489 17578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:18.417855 17578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:18.418049 17578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:18.425530 17578 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:34633
I20260923 18:15:18.425563 17645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:34633 every 8 connection(s)
I20260923 18:15:18.426601 17578 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:18.429646 17646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:18.433272 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 17578
I20260923 18:15:18.433432 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:18.433681 27951 external_mini_cluster.cc:1428] Setting key 38ceea8bcde5f3b2a432177214134f4f
I20260923 18:15:18.436021 17646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5: Bootstrap starting.
I20260923 18:15:18.438483 17646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:18.439445 17646 log.cc:824] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:18.441880 17646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5: No bootstrap required, opened a new log
I20260923 18:15:18.443275 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:18.435168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42631 (local address 127.27.75.254:34633)
0923 18:15:18.435598 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:18.435616 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:18.435653 (+ 37us) server_negotiation.cc:407] Connection header received
0923 18:15:18.436327 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:18.436346 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:18.436716 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:18.437008 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:18.437459 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:18.438480 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:18.439780 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:18.440318 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:18.440466 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:18.441410 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:18.441416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:18.441431 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:18.441901 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:18.441986 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:18.442022 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:18.442120 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:18.442292 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 18:15:18.442526 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":118,"threads_started":1}
I20260923 18:15:18.445645 17646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf96e5e7d84d5ab10340baf196ceb5" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34633 } }
I20260923 18:15:18.445892 17646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:18.445938 17646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cf96e5e7d84d5ab10340baf196ceb5, State: Initialized, Role: FOLLOWER
I20260923 18:15:18.446157 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:18.446449 17646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf96e5e7d84d5ab10340baf196ceb5" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34633 } }
I20260923 18:15:18.446604 17646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:18.446671 17646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:18.446768 17646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:18.448031 17646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf96e5e7d84d5ab10340baf196ceb5" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34633 } }
I20260923 18:15:18.448421 17646 leader_election.cc:302] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25cf96e5e7d84d5ab10340baf196ceb5; no voters:
I20260923 18:15:18.448848 17646 leader_election.cc:288] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:18.448937 17656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:18.449180 17656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [term 1 LEADER]: Becoming Leader. State: Replica: 25cf96e5e7d84d5ab10340baf196ceb5, State: Running, Role: LEADER
I20260923 18:15:18.449571 17656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf96e5e7d84d5ab10340baf196ceb5" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34633 } }
I20260923 18:15:18.450237 17646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:18.451357 17658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25cf96e5e7d84d5ab10340baf196ceb5. Latest consensus state: current_term: 1 leader_uuid: "25cf96e5e7d84d5ab10340baf196ceb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf96e5e7d84d5ab10340baf196ceb5" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34633 } } }
I20260923 18:15:18.451387 17657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25cf96e5e7d84d5ab10340baf196ceb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cf96e5e7d84d5ab10340baf196ceb5" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 34633 } } }
I20260923 18:15:18.451500 17658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:18.451503 17657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:18.453473 17661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:18.453670 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:18.448433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38851 (local address 127.27.75.254:34633)
0923 18:15:18.448562 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:18.448567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:18.448716 (+ 149us) server_negotiation.cc:407] Connection header received
0923 18:15:18.448810 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:18.448812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:18.448866 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:18.448949 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:18.449392 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:18.450113 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:18.451314 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:18.451650 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:18.451730 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:18.453194 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:18.453197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:18.453200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:18.453213 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:18.453244 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:18.453258 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:18.453337 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:18.453469 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 18:15:18.453531 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:15:18.456761 17661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:18.462357 17661 catalog_manager.cc:1347] Generated new cluster ID: 34468b9e04e44bd8a0ef663fd73aeb33
I20260923 18:15:18.462443 17661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:18.487692 17661 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:18.489070 17661 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:18.500164 17661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25cf96e5e7d84d5ab10340baf196ceb5: Generated new TSK 0
I20260923 18:15:18.500996 17661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:18.515175 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 17578
2026-09-23T18:15:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:18 dist-test-slave-wsx5 krb5kdc[17677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:18 dist-test-slave-wsx5 krb5kdc[17677](info): set up 2 sockets
Sep 23 18:15:18 dist-test-slave-wsx5 krb5kdc[17677](info): commencing operation
krb5kdc: starting...
W20260923 18:15:19.459353 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:19 dist-test-slave-wsx5 krb5kdc[17677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:15:19.588837 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:37139
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:35327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:37139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:19.702178 17690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:19.702476 17690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:19.702517 17690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:19.706787 17690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:19.706854 17690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:19.706872 17690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:19.706888 17690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:19.706902 17690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:19.712414 17690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:35327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:37139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:37139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:19.713551 17690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:19.714576 17690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:15:19.723477 17690 server_base.cc:1034] running on GCE node
W20260923 18:15:19.723444 17695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:19.723472 17698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:19.723726 17696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:19.724324 17690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:19.725562 17690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:19.726719 17690 hybrid_clock.cc:648] HybridClock initialized: now 1790187319726692 us; error 51 us; skew 500 ppm
Sep 23 18:15:19 dist-test-slave-wsx5 krb5kdc[17677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187319, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:19.730286 17690 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:15:19.731444 17690 webserver.cc:460] Webserver started at http://127.27.75.254:33947/ using document root <none> and password file <none>
I20260923 18:15:19.732147 17690 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:19.732234 17690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:19.732496 17690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:19.735165 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "c7387723f6e6493e84f95eb1c91c0b8a"
format_stamp: "Formatted at 2026-09-23 18:15:19 on dist-test-slave-wsx5"
server_key: "2486810d4c576184d5cedbdbcc3eb77e"
server_key_iv: "15556979f463a6fd0db020593ef6c2e9"
server_key_version: "encryptionkey@0"
I20260923 18:15:19.735716 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "c7387723f6e6493e84f95eb1c91c0b8a"
format_stamp: "Formatted at 2026-09-23 18:15:19 on dist-test-slave-wsx5"
server_key: "2486810d4c576184d5cedbdbcc3eb77e"
server_key_iv: "15556979f463a6fd0db020593ef6c2e9"
server_key_version: "encryptionkey@0"
I20260923 18:15:19.739910 17690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 18:15:19.742755 17705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:19.744093 17690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 18:15:19.744223 17690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "c7387723f6e6493e84f95eb1c91c0b8a"
format_stamp: "Formatted at 2026-09-23 18:15:19 on dist-test-slave-wsx5"
server_key: "2486810d4c576184d5cedbdbcc3eb77e"
server_key_iv: "15556979f463a6fd0db020593ef6c2e9"
server_key_version: "encryptionkey@0"
I20260923 18:15:19.744339 17690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:19.749078 17690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:19.752280 17690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:19.752466 17690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:19.759253 17690 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:37139
I20260923 18:15:19.759305 17757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:37139 every 8 connection(s)
I20260923 18:15:19.760229 17690 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:19.763331 17758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:19.765321 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 17690
I20260923 18:15:19.765458 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:19.765738 27951 external_mini_cluster.cc:1428] Setting key 0eacab27667d4baeffe4f1f1e6149d54
I20260923 18:15:19.770387 17758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a: Bootstrap starting.
I20260923 18:15:19.771014 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:19.767191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54791 (local address 127.27.75.254:37139)
0923 18:15:19.767626 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:19.767641 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:19.767676 (+ 35us) server_negotiation.cc:407] Connection header received
0923 18:15:19.768326 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:19.768344 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:19.768646 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:19.768966 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:19.769756 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:19.769760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:19.769773 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:19.769784 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:19.769840 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:19.769861 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:19.769991 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:19.770062 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 18:15:19.770226 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260923 18:15:19.772998 17758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:19.773851 17758 log.cc:824] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:19.773944 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:19.775933 17758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a: No bootstrap required, opened a new log
I20260923 18:15:19.777508 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:19.775896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55685 (local address 127.27.75.254:37139)
0923 18:15:19.776006 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:19.776011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:19.776204 (+ 193us) server_negotiation.cc:407] Connection header received
0923 18:15:19.776332 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:19.776335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:19.776369 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:19.776432 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:19.777114 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:19.777146 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:19.777149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 18:15:19.777161 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:19.777189 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 18:15:19.777201 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:19.777274 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:19.777347 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 18:15:19.777388 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 18:15:19.779747 17758 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7387723f6e6493e84f95eb1c91c0b8a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37139 } }
I20260923 18:15:19.779994 17758 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 18:15:19.780041 17758 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7387723f6e6493e84f95eb1c91c0b8a, State: Initialized, Role: FOLLOWER
I20260923 18:15:19.780473 17758 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7387723f6e6493e84f95eb1c91c0b8a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37139 } }
I20260923 18:15:19.780606 17758 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 18:15:19.780669 17758 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 18:15:19.780764 17758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:19.781742 17758 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7387723f6e6493e84f95eb1c91c0b8a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37139 } }
I20260923 18:15:19.782073 17758 leader_election.cc:302] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7387723f6e6493e84f95eb1c91c0b8a; no voters:
I20260923 18:15:19.782382 17758 leader_election.cc:288] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 18:15:19.782472 17768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:19.782666 17768 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [term 1 LEADER]: Becoming Leader. State: Replica: c7387723f6e6493e84f95eb1c91c0b8a, State: Running, Role: LEADER
I20260923 18:15:19.782989 17768 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7387723f6e6493e84f95eb1c91c0b8a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37139 } }
I20260923 18:15:19.783762 17758 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:19.784933 17769 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7387723f6e6493e84f95eb1c91c0b8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7387723f6e6493e84f95eb1c91c0b8a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37139 } } }
I20260923 18:15:19.784950 17770 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7387723f6e6493e84f95eb1c91c0b8a. Latest consensus state: current_term: 1 leader_uuid: "c7387723f6e6493e84f95eb1c91c0b8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7387723f6e6493e84f95eb1c91c0b8a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37139 } } }
I20260923 18:15:19.785096 17770 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:19.785094 17769 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:19.785542 17777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:19.788355 17777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 18:15:19.794103 17777 catalog_manager.cc:1347] Generated new cluster ID: dde14e990012407ea4b9d729c8277d7e
I20260923 18:15:19.794193 17777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:19.804679 17777 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:19.805015 17777 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:19.811471 17777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7387723f6e6493e84f95eb1c91c0b8a: Generated new TSK 0
I20260923 18:15:19.812361 17777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 18:15:19.826644 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 17690
2026-09-23T18:15:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1311 ms)
[----------] 18 tests from AuthTokenIssuingTest (24243 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:19 dist-test-slave-wsx5 krb5kdc[17789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:19 dist-test-slave-wsx5 krb5kdc[17789](info): set up 2 sockets
Sep 23 18:15:19 dist-test-slave-wsx5 krb5kdc[17789](info): commencing operation
krb5kdc: starting...
W20260923 18:15:20.791544 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:20 dist-test-slave-wsx5 krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:15:20.910355 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:37575
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:37575,127.27.75.253:46207,127.27.75.252:43121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:21.024278 17802 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:15:21.024617 17802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:21.024641 17802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:21.024688 17802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:21.029702 17802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:21.029801 17802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:21.029821 17802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:21.029838 17802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:21.029853 17802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:21.036118 17802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:37575,127.27.75.253:46207,127.27.75.252:43121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:37575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:21.037457 17802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:21.038597 17802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:21.047699 17807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:21.047773 17808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:21.047672 17810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:21.047919 17802 server_base.cc:1034] running on GCE node
I20260923 18:15:21.048527 17802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:21.049793 17802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:21.050995 17802 hybrid_clock.cc:648] HybridClock initialized: now 1790187321050976 us; error 36 us; skew 500 ppm
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:21.054041 17802 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:15:21.055033 17802 webserver.cc:460] Webserver started at http://127.27.75.254:43645/ using document root <none> and password file <none>
I20260923 18:15:21.055577 17802 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:21.055642 17802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:21.055842 17802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:21.058007 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a"
format_stamp: "Formatted at 2026-09-23 18:15:21 on dist-test-slave-wsx5"
server_key: "8f3a872e0b46409e6f8b33c7b2eb46db"
server_key_iv: "725f56ca6687ca996809d482e9588b1c"
server_key_version: "encryptionkey@0"
I20260923 18:15:21.058548 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a"
format_stamp: "Formatted at 2026-09-23 18:15:21 on dist-test-slave-wsx5"
server_key: "8f3a872e0b46409e6f8b33c7b2eb46db"
server_key_iv: "725f56ca6687ca996809d482e9588b1c"
server_key_version: "encryptionkey@0"
I20260923 18:15:21.062312 17802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260923 18:15:21.064849 17817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:21.066211 17802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 18:15:21.066351 17802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a"
format_stamp: "Formatted at 2026-09-23 18:15:21 on dist-test-slave-wsx5"
server_key: "8f3a872e0b46409e6f8b33c7b2eb46db"
server_key_iv: "725f56ca6687ca996809d482e9588b1c"
server_key_version: "encryptionkey@0"
I20260923 18:15:21.066454 17802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:21.079083 17802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:21.082249 17802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:21.082440 17802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:21.089277 17802 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:37575
I20260923 18:15:21.089541 17870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:37575 every 8 connection(s)
I20260923 18:15:21.090245 17802 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:21.092957 17871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:21.096683 17871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:21.097163 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 17802
I20260923 18:15:21.097285 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:21.097568 27951 external_mini_cluster.cc:1428] Setting key a510ad04216c6ab445a119ed98c16cf1
WARNING: no policy specified for kudu/127.27.75.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.253@KRBTEST.COM" created.
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:15:21.114202 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.099420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:37575 (local address 127.0.0.1:54182)
0923 18:15:21.100036 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.100084 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:21.100944 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:21.101651 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:21.101656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:21.102095 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:21.102347 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.102362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.103622 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.103626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.104828 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.104843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.105646 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.105650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.105817 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.106398 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:21.106416 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:21.108946 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:21.110990 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.110999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.111012 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.111247 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.111579 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.111582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.111584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.111691 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.112301 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:21.112308 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:21.112443 (+ 135us) client_negotiation.cc:769] Sending connection context
0923 18:15:21.112602 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 18:15:21.113033 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":94,"spinlock_wait_cycles":52992,"thread_start_us":152,"threads_started":1}
I20260923 18:15:21.114202 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.099420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.27.75.254:37575)
0923 18:15:21.099907 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.099925 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.100197 (+ 272us) server_negotiation.cc:407] Connection header received
0923 18:15:21.101283 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.101288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.101471 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.101632 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.102490 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.103442 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.104984 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.105452 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.105585 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.109162 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.109171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.109188 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.109218 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.110870 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.111362 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.111367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.111368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.111482 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.111773 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.111776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.111778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.112183 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.112301 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.112692 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.112944 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":140,"threads_started":1}
I20260923 18:15:21.116863 17871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:21.117786 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.117451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:46207 (local address 127.0.0.1:43546)
0923 18:15:21.117588 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.117673 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:46207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 18:15:21.118016 17818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.75.253:46207: connect: Connection refused (error 111)
I20260923 18:15:21.118986 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.118749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:46207 (local address 127.0.0.1:43552)
0923 18:15:21.118850 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.118921 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:46207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 18:15:21.119165 17871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.253:46207: Network error: Client connection negotiation failed: client connection to 127.27.75.253:46207: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab.
Entry for principal kudu/127.27.75.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab.
I20260923 18:15:21.124369 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.253:46207
--webserver_interface=127.27.75.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab
--principal=kudu/127.27.75.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:37575,127.27.75.253:46207,127.27.75.252:43121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:15:21.168509 17871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } attempt: 1
I20260923 18:15:21.169483 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.169111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:46207 (local address 127.0.0.1:43558)
0923 18:15:21.169333 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.169401 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:46207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 18:15:21.170220 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.170004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:46207 (local address 127.0.0.1:43572)
0923 18:15:21.170096 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.170150 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:46207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 18:15:21.170341 17871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.253:46207: Network error: Client connection negotiation failed: client connection to 127.27.75.253:46207: connect: Connection refused (error 111)
W20260923 18:15:21.236387 17878 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:15:21.236706 17878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:21.236783 17878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:21.236850 17878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 18:15:21.238503 17871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } attempt: 2
I20260923 18:15:21.239279 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.238987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:46207 (local address 127.0.0.1:43578)
0923 18:15:21.239130 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.239192 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:46207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 18:15:21.239988 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.239776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:46207 (local address 127.0.0.1:43594)
0923 18:15:21.239879 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.239927 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:46207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 18:15:21.240092 17871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.253:46207: Network error: Client connection negotiation failed: client connection to 127.27.75.253:46207: connect: Connection refused (error 111)
W20260923 18:15:21.241254 17878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:21.241310 17878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:21.241338 17878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:21.241371 17878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:21.241400 17878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:21.246918 17878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:37575,127.27.75.253:46207,127.27.75.252:43121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab
--principal=kudu/127.27.75.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.253:46207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.75.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:21.248034 17878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:21.249042 17878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:15:21.257448 17878 server_base.cc:1034] running on GCE node
W20260923 18:15:21.257388 17883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:21.257403 17886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:21.257388 17884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:21.258337 17878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:21.259460 17878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:21.260648 17878 hybrid_clock.cc:648] HybridClock initialized: now 1790187321260607 us; error 56 us; skew 500 ppm
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:21.263576 17878 init.cc:377] Logged in from keytab as kudu/127.27.75.253@KRBTEST.COM (short username kudu)
I20260923 18:15:21.264567 17878 webserver.cc:460] Webserver started at http://127.27.75.253:38591/ using document root <none> and password file <none>
I20260923 18:15:21.265174 17878 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:21.265259 17878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:21.265476 17878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:21.267625 17878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/data/instance:
uuid: "fa1189ae2b0c40eba3886eff1a595168"
format_stamp: "Formatted at 2026-09-23 18:15:21 on dist-test-slave-wsx5"
server_key: "1c08fe3f957a2f6ba113da4a8071d084"
server_key_iv: "7b9d271355f7788d6935b22ad2112a4a"
server_key_version: "encryptionkey@0"
I20260923 18:15:21.268210 17878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/wal/instance:
uuid: "fa1189ae2b0c40eba3886eff1a595168"
format_stamp: "Formatted at 2026-09-23 18:15:21 on dist-test-slave-wsx5"
server_key: "1c08fe3f957a2f6ba113da4a8071d084"
server_key_iv: "7b9d271355f7788d6935b22ad2112a4a"
server_key_version: "encryptionkey@0"
I20260923 18:15:21.272341 17878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260923 18:15:21.275118 17893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:21.276552 17878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 18:15:21.276741 17878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/wal
uuid: "fa1189ae2b0c40eba3886eff1a595168"
format_stamp: "Formatted at 2026-09-23 18:15:21 on dist-test-slave-wsx5"
server_key: "1c08fe3f957a2f6ba113da4a8071d084"
server_key_iv: "7b9d271355f7788d6935b22ad2112a4a"
server_key_version: "encryptionkey@0"
I20260923 18:15:21.276865 17878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:21.289469 17878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:21.292774 17878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:21.292956 17878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:21.299767 17878 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.253:46207
I20260923 18:15:21.300047 17946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.253:46207 every 8 connection(s)
I20260923 18:15:21.300776 17878 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
I20260923 18:15:21.303645 17947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:21.307544 17947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:21.310948 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 17878
I20260923 18:15:21.311074 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-1/wal/instance
I20260923 18:15:21.311347 27951 external_mini_cluster.cc:1428] Setting key 3622d415bf5005418b39f060aa5bfaae
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.75.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.252@KRBTEST.COM" created.
I20260923 18:15:21.321388 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.309523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.27.75.254:37575)
0923 18:15:21.309652 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.309655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.310577 (+ 922us) server_negotiation.cc:407] Connection header received
0923 18:15:21.311555 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.311559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.311625 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.311732 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.313093 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.313670 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.314856 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.315105 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.315198 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.317659 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.317662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.317664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.317691 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.319023 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.319649 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.319652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.319653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.319737 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.320086 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.320089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.320090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.320311 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.320403 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.321065 (+ 662us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.321224 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 18:15:21.322220 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.309900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:37575 (local address 127.0.0.1:54184)
0923 18:15:21.310406 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.310453 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:21.311319 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:21.311975 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:21.311982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:21.312359 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:21.312902 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.312917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.313810 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.313813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.314715 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.314725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.315256 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.315260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.315432 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.316006 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:21.316023 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:21.317472 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:21.319144 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.319150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.319165 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.319525 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.319825 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.319827 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.319828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.320012 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.320417 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:21.320424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:21.320774 (+ 350us) client_negotiation.cc:769] Sending connection context
0923 18:15:21.320974 (+ 200us) client_negotiation.cc:240] Negotiation successful
0923 18:15:21.321291 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260923 18:15:21.323606 17947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:21.331265 17871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } attempt: 3
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
I20260923 18:15:21.336200 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.324120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:46207 (local address 127.0.0.1:43608)
0923 18:15:21.324279 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.324292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:21.324364 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:21.324928 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:21.324931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:21.325107 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:21.325249 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.325257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.326211 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.326215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.327068 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.327080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.327680 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.327684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.327794 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.328272 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:21.328286 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:21.332281 (+ 3995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:21.334750 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.334753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.334755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.334941 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.335224 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.335227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.335229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.335306 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.335726 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:21.335729 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:21.335813 (+ 84us) client_negotiation.cc:769] Sending connection context
0923 18:15:21.335910 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 18:15:21.336032 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260923 18:15:21.336853 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.324256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.27.75.253:46207)
0923 18:15:21.324569 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.324578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.324597 (+ 19us) server_negotiation.cc:407] Connection header received
0923 18:15:21.324631 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.324635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.324798 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.324911 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.325354 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.326085 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.327200 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.327564 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.327648 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.332451 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.332465 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.332484 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.332509 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.334655 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.335053 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.335060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.335061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.335146 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.335378 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.335381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.335383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.335625 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.335755 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.336023 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.336155 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":131,"threads_started":1}
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab.
Entry for principal kudu/127.27.75.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab.
I20260923 18:15:21.338739 17947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:21.339524 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.252:43121
--webserver_interface=127.27.75.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab
--principal=kudu/127.27.75.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:37575,127.27.75.253:46207,127.27.75.252:43121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:15:21.341288 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.340894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49232)
0923 18:15:21.341066 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.341169 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260923 18:15:21.341512 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.331928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:46207 (local address 127.0.0.1:43624)
0923 18:15:21.332144 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.332158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:21.332256 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:21.332618 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:21.332621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:21.332812 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:21.332958 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.332967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.334251 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.334254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.335280 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.335292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.336269 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.336273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.336373 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.336838 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:21.336851 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:21.338204 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:21.340092 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.340095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.340097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.340290 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.340530 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.340533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.340535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.340612 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.340948 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:21.340951 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:21.341047 (+ 96us) client_negotiation.cc:769] Sending connection context
0923 18:15:21.341196 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 18:15:21.341351 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
W20260923 18:15:21.341516 17894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
I20260923 18:15:21.341511 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.332000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.27.75.253:46207)
0923 18:15:21.332370 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.332378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.332392 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:15:21.332430 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.332432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.332490 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.332572 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.333108 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.334110 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.335791 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.336116 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.336215 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.338352 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.338356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.338357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.338378 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.339992 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.340374 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.340378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.340379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.340452 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.340687 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.340690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.340692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.340864 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.340943 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.341252 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.341352 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":131,"threads_started":1}
I20260923 18:15:21.342599 17871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:21.342739 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.342325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49242)
0923 18:15:21.342620 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.342671 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
W20260923 18:15:21.342888 17947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:43121: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
I20260923 18:15:21.343236 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.343015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49256)
0923 18:15:21.343125 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.343173 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 18:15:21.343909 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.343681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49262)
0923 18:15:21.343811 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.343854 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260923 18:15:21.344033 17871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:43121: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
I20260923 18:15:21.375216 17871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } attempt: 1
I20260923 18:15:21.376109 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.375774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49266)
0923 18:15:21.375948 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.376018 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 18:15:21.376915 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.376677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49282)
0923 18:15:21.376806 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.376852 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 18:15:21.377066 17871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:43121: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
I20260923 18:15:21.392194 17947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } attempt: 1
I20260923 18:15:21.393072 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.392780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49284)
0923 18:15:21.392926 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.392993 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 18:15:21.393815 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.393610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49286)
0923 18:15:21.393711 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.393758 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 18:15:21.393944 17947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:43121: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
I20260923 18:15:21.452322 17871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } attempt: 2
W20260923 18:15:21.452858 17954 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:15:21.453183 17954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:21.453234 17954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:21.453287 17954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 18:15:21.453279 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.452924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49290)
0923 18:15:21.453077 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.453183 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 18:15:21.454100 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.453876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49296)
0923 18:15:21.453990 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.454040 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 18:15:21.454331 17871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:43121: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
W20260923 18:15:21.457952 17954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:21.458019 17954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:21.458037 17954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:21.458053 17954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:21.458076 17954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:21.462116 17947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } attempt: 2
I20260923 18:15:21.463045 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.462693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49312)
0923 18:15:21.462878 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.462960 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 18:15:21.463815 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.463548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49314)
0923 18:15:21.463685 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.463742 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 18:15:21.463984 17954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:37575,127.27.75.253:46207,127.27.75.252:43121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab
--principal=kudu/127.27.75.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.252:43121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.75.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
W20260923 18:15:21.464150 17947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:43121: Network error: Client connection negotiation failed: client connection to 127.27.75.252:43121: connect: Connection refused (error 111)
I20260923 18:15:21.465196 17954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:21.466269 17954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 18:15:21.475258 17954 server_base.cc:1034] running on GCE node
W20260923 18:15:21.475204 17960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:21.475204 17961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:21.475250 17963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:21.476151 17954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:21.477396 17954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:21.478607 17954 hybrid_clock.cc:648] HybridClock initialized: now 1790187321478583 us; error 38 us; skew 500 ppm
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:21.481765 17954 init.cc:377] Logged in from keytab as kudu/127.27.75.252@KRBTEST.COM (short username kudu)
I20260923 18:15:21.482738 17954 webserver.cc:460] Webserver started at http://127.27.75.252:45715/ using document root <none> and password file <none>
I20260923 18:15:21.483291 17954 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:21.483369 17954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:21.483572 17954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:21.485746 17954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/data/instance:
uuid: "12b95af4521a40cf961adbed04833b09"
format_stamp: "Formatted at 2026-09-23 18:15:21 on dist-test-slave-wsx5"
server_key: "e7423a89bfc3471c5604930accb69bb9"
server_key_iv: "a93e01cd99401e816f31b9b4849115b1"
server_key_version: "encryptionkey@0"
I20260923 18:15:21.486318 17954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/wal/instance:
uuid: "12b95af4521a40cf961adbed04833b09"
format_stamp: "Formatted at 2026-09-23 18:15:21 on dist-test-slave-wsx5"
server_key: "e7423a89bfc3471c5604930accb69bb9"
server_key_iv: "a93e01cd99401e816f31b9b4849115b1"
server_key_version: "encryptionkey@0"
I20260923 18:15:21.490172 17954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 18:15:21.492976 17970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:21.494445 17954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 18:15:21.494581 17954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/wal
uuid: "12b95af4521a40cf961adbed04833b09"
format_stamp: "Formatted at 2026-09-23 18:15:21 on dist-test-slave-wsx5"
server_key: "e7423a89bfc3471c5604930accb69bb9"
server_key_iv: "a93e01cd99401e816f31b9b4849115b1"
server_key_version: "encryptionkey@0"
I20260923 18:15:21.494705 17954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:21.512796 17954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:21.515878 17954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:21.516074 17954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:21.522939 17954 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.252:43121
I20260923 18:15:21.523242 18023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.252:43121 every 8 connection(s)
I20260923 18:15:21.523921 17954 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
I20260923 18:15:21.526489 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 17954
I20260923 18:15:21.526611 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790187042487232-27951-0/minicluster-data/master-2/wal/instance
I20260923 18:15:21.526862 18024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:21.526932 27951 external_mini_cluster.cc:1428] Setting key cd6810a395e96d367c2eb920e69cb193
I20260923 18:15:21.531792 18024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:15:21.542526 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.528284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.27.75.252:43121)
0923 18:15:21.528678 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.528693 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.528728 (+ 35us) server_negotiation.cc:407] Connection header received
0923 18:15:21.529585 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.529604 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.529940 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.530217 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.531115 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.532250 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.533216 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.533788 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.533948 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.537061 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.537066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.537076 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.537105 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.539074 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.539562 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.539568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.539575 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.539696 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.540009 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.540012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.540013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.540545 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.540718 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.541145 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.541455 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260923 18:15:21.544438 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.534099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.27.75.254:37575)
0923 18:15:21.534258 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.534263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.534575 (+ 312us) server_negotiation.cc:407] Connection header received
0923 18:15:21.534766 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.534769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.534819 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.534882 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.535746 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.536510 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.537620 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.537932 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.538047 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.540300 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.540304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.540307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.540358 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.542178 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.542874 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.542878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.542880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.542982 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.543307 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.543310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.543312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.543560 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.543658 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.544162 (+ 504us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.544288 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 18:15:21.544497 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.534014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:37575 (local address 127.0.0.1:54196)
0923 18:15:21.534458 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.534504 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:21.534636 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:21.534972 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:21.534979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:21.535442 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:21.535620 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.535628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.536651 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.536655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.537501 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.537513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.538053 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.538058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.538189 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.538695 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:21.538708 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:21.540155 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:21.542344 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.542352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.542357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.542587 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.543121 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.543123 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.543125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.543212 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.543652 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:21.543659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:21.543917 (+ 258us) client_negotiation.cc:769] Sending connection context
0923 18:15:21.544083 (+ 166us) client_negotiation.cc:240] Negotiation successful
0923 18:15:21.544313 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":149,"threads_started":1}
I20260923 18:15:21.545723 18024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
I20260923 18:15:21.553493 17871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } attempt: 3
I20260923 18:15:21.555060 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.546320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.27.75.253:46207)
0923 18:15:21.546472 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.546476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.546500 (+ 24us) server_negotiation.cc:407] Connection header received
0923 18:15:21.546623 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.546626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.546674 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.546733 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.547220 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.548005 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.549175 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.549537 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.549645 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.551803 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.551806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.551809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.551840 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.553157 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.553570 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.553573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.553575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.553664 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.554019 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.554023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.554024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.554275 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.554379 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.554802 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.554931 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 18:15:21.555284 17947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } attempt: 3
I20260923 18:15:21.555398 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.545531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.27.75.254:37575)
0923 18:15:21.545695 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.545698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.545710 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:15:21.545811 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.545814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.545858 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.545923 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.546576 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.547321 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.548423 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.548770 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.548878 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.551484 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.551488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.551491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.551521 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.553043 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.553762 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.553765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.553767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.553845 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.554138 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.554141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.554143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.554355 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.554581 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.554682 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.555255 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 18:15:21.555527 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.546175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:46207 (local address 127.0.0.1:43638)
0923 18:15:21.546414 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.546434 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:21.546518 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:21.546766 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:21.546769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:21.546947 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:21.547061 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.547068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.548169 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.548172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.549016 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.549026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.549671 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.549674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.549751 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.550153 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:21.550166 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:21.551651 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:21.553277 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.553279 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.553281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.553479 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.553742 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.553746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.553748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.553843 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.554424 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:21.554427 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:21.554527 (+ 100us) client_negotiation.cc:769] Sending connection context
0923 18:15:21.555249 (+ 722us) client_negotiation.cc:240] Negotiation successful
0923 18:15:21.555384 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260923 18:15:21.556689 18024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
Sep 23 18:15:21 dist-test-slave-wsx5 krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
I20260923 18:15:21.564952 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.554041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49338)
0923 18:15:21.554193 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.554207 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:21.554283 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:21.554720 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:21.554723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:21.554934 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:21.555025 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.555030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.556576 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.556580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.557389 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.557398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.557991 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.557995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.558107 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.558612 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:21.558627 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:21.560106 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:21.562716 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.562719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.562721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.562931 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.563269 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.563273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.563275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.563375 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.563837 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:21.563841 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:21.564449 (+ 608us) client_negotiation.cc:769] Sending connection context
0923 18:15:21.564581 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 18:15:21.564758 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 18:15:21.565634 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.554398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.27.75.252:43121)
0923 18:15:21.554515 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.554519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.554529 (+ 10us) server_negotiation.cc:407] Connection header received
0923 18:15:21.554558 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.554562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.554600 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.554823 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.555129 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.556448 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.557513 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.557868 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.560228 (+ 2360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.560335 (+ 107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.560339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.560342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.560359 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.562531 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.563048 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.563051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.563053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.563138 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.563484 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.563487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.563489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.563709 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.563834 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.565354 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.565480 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:15:21.566905 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.556836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.27.75.253:46207)
0923 18:15:21.556928 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.556932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.557172 (+ 240us) server_negotiation.cc:407] Connection header received
0923 18:15:21.557552 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.557556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.557597 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.557646 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.558547 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.559265 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.560406 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.560702 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.560773 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.563511 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.563515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.563518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.563547 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.565079 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.565612 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.565616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.565617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.565686 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.566000 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.566004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.566005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.566247 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.566465 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.566629 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.566752 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37,"spinlock_wait_cycles":768}
I20260923 18:15:21.567700 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.555641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49346)
0923 18:15:21.555774 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.555788 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:21.555854 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:21.556623 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:21.556626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:21.556808 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:21.556905 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.556912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.558630 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.558633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.559416 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.559427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.560122 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.560126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.560212 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.560686 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:21.560700 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:21.563614 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:21.565555 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.565559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.565561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.565806 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.566074 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.566077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.566079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.566169 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.567165 (+ 996us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:21.567168 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:21.567269 (+ 101us) client_negotiation.cc:769] Sending connection context
0923 18:15:21.567341 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:15:21.567546 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"spinlock_wait_cycles":3456}
I20260923 18:15:21.568300 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.556081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.27.75.252:43121)
0923 18:15:21.556402 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.556410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.556423 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:15:21.556453 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.556456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.556496 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.557013 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.557098 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.558503 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.559574 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.559992 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.560420 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.563772 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.563776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.563778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.563800 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.565435 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.565916 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.565920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.565921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.566002 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.566255 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.566259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.566260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.567062 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.567975 (+ 913us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.568049 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.568154 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":256,"thread_start_us":134,"threads_started":1}
I20260923 18:15:21.568742 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.557222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:43121 (local address 127.0.0.1:49348)
0923 18:15:21.557315 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:21.557328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:21.557391 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:21.557710 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:21.557713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:21.557887 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:21.557982 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.557989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.559219 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.559222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.560011 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:21.560022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.560583 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.560586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:21.560669 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.561055 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:21.561068 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:21.563248 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:21.566739 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.566742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.566744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.566961 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.567360 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:21.567363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:21.567365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:21.567533 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:21.567992 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:21.567995 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:21.568077 (+ 82us) client_negotiation.cc:769] Sending connection context
0923 18:15:21.568149 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 18:15:21.568281 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":1}
I20260923 18:15:21.569597 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.557249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.27.75.252:43121)
0923 18:15:21.557518 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.557532 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.557543 (+ 11us) server_negotiation.cc:407] Connection header received
0923 18:15:21.557570 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.557573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.557607 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.558080 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.558194 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.559113 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.560141 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.560488 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.560866 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.563378 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.563382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.563385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.563409 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.566643 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.567171 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.567174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.567175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.567265 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.567662 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.567666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.567668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.567883 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.569024 (+ 1141us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.569156 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.569292 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
I20260923 18:15:21.570561 17871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a: Bootstrap starting.
I20260923 18:15:21.572269 18024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09: Bootstrap starting.
I20260923 18:15:21.572486 17947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168: Bootstrap starting.
I20260923 18:15:21.573743 17871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:21.574806 18024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:21.575016 17947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:21.575388 17871 log.cc:824] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:21.575707 18024 log.cc:824] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:21.575801 17947 log.cc:824] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:21.575903 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:21.577742 17871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a: No bootstrap required, opened a new log
I20260923 18:15:21.577746 17947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168: No bootstrap required, opened a new log
I20260923 18:15:21.577358 17844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54212
I20260923 18:15:21.578855 18024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09: No bootstrap required, opened a new log
I20260923 18:15:21.580804 17871 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } }
I20260923 18:15:21.581060 17871 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f2f1a94de04c28ab35e8d5e2ce038a, State: Initialized, Role: FOLLOWER
I20260923 18:15:21.581518 17871 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } }
I20260923 18:15:21.581751 17947 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } }
I20260923 18:15:21.581990 17947 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1189ae2b0c40eba3886eff1a595168, State: Initialized, Role: FOLLOWER
I20260923 18:15:21.582319 17947 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } }
I20260923 18:15:21.583037 17871 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:21.583153 17844 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 18:15:21.583272 17844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:21.583328 18024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } }
I20260923 18:15:21.583575 18024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b95af4521a40cf961adbed04833b09, State: Initialized, Role: FOLLOWER
I20260923 18:15:21.583658 17947 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:21.583894 18024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } }
W20260923 18:15:21.584019 17823 tablet.cc:2369] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 18:15:21.584625 17844 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } }
I20260923 18:15:21.585191 18024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:21.586884 17998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "12b95af4521a40cf961adbed04833b09"
I20260923 18:15:21.586884 17919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fa1189ae2b0c40eba3886eff1a595168"
I20260923 18:15:21.587107 17919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:21.586006 17844 leader_election.cc:288] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [CANDIDATE]: Term 1 election: Requested vote from peers fa1189ae2b0c40eba3886eff1a595168 (127.27.75.253:46207), 12b95af4521a40cf961adbed04833b09 (127.27.75.252:43121)
I20260923 18:15:21.587107 17998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:21.588474 17919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f2f1a94de04c28ab35e8d5e2ce038a in term 1.
I20260923 18:15:21.588809 17998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f2f1a94de04c28ab35e8d5e2ce038a in term 1.
I20260923 18:15:21.589488 17818 leader_election.cc:302] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6f2f1a94de04c28ab35e8d5e2ce038a, fa1189ae2b0c40eba3886eff1a595168; no voters:
W20260923 18:15:21.589895 18056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:15:21.589957 18056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:15:21.582890 18035 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } } }
I20260923 18:15:21.593215 18035 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:21.595305 18071 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } } }
I20260923 18:15:21.595484 18071 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:21.595573 18071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:21.595644 18071 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [term 1 LEADER]: Becoming Leader. State: Replica: f6f2f1a94de04c28ab35e8d5e2ce038a, State: Running, Role: LEADER
I20260923 18:15:21.595947 18071 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } }
I20260923 18:15:21.596799 17954 master_runner.cc:263] No leader master found from master 12b95af4521a40cf961adbed04833b09
I20260923 18:15:21.596876 17954 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:15:21.597831 18071 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f2f1a94de04c28ab35e8d5e2ce038a. Latest consensus state: current_term: 1 leader_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } } }
I20260923 18:15:21.597946 18071 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:21.597982 17878 master_runner.cc:263] No leader master found from master fa1189ae2b0c40eba3886eff1a595168
I20260923 18:15:21.598048 17878 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:15:21.598511 18078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:21.600755 17802 master_runner.cc:263] No leader master found from master f6f2f1a94de04c28ab35e8d5e2ce038a
I20260923 18:15:21.600819 17802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:15:21.601141 18078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260923 18:15:21.593075 18059 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:15:21.603329 18059 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:15:21.610996 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.601554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.27.75.253:46207)
0923 18:15:21.601682 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.601687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.602050 (+ 363us) server_negotiation.cc:407] Connection header received
0923 18:15:21.602252 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.602256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.602306 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.602354 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.603431 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.604066 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.605226 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.605550 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.605627 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.607414 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.607418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.607422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.607452 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.609037 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.609514 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.609518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.609520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.609613 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.609942 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.609945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.609947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.610192 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.610319 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.610690 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.610814 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 18:15:21.613014 18037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } } }
I20260923 18:15:21.613193 18037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:21.614013 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.602103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.27.75.252:43121)
0923 18:15:21.602209 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.602213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.603694 (+ 1481us) server_negotiation.cc:407] Connection header received
0923 18:15:21.603787 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.603790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.603837 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.603911 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.604768 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.605556 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.606595 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.606986 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.607067 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.609113 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.609144 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.609147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.609176 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.610867 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.612410 (+ 1543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.612414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.612416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.612523 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.612872 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.612876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.612877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.613101 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.613254 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.613741 (+ 487us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.613866 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 18:15:21.615698 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:21.602795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.27.75.254:37575)
0923 18:15:21.602901 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.602905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.603949 (+ 1044us) server_negotiation.cc:407] Connection header received
0923 18:15:21.604093 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.604096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.604140 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.604183 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.605170 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.605956 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.607061 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:21.607412 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:21.607479 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:21.612228 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:21.612232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:21.612235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:21.612268 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:21.613909 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.614354 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.614357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.614359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.614446 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:21.614727 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:21.614731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:21.614733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:21.614969 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:21.615078 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:21.615425 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 18:15:21.615545 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 18:15:21.619352 17998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [term 1 FOLLOWER]: Refusing update from remote peer f6f2f1a94de04c28ab35e8d5e2ce038a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:15:21.619772 17919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [term 1 FOLLOWER]: Refusing update from remote peer f6f2f1a94de04c28ab35e8d5e2ce038a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260923 18:15:21.620261 18069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:15:21.620352 18069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:15:21.620779 18071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:15:21.621227 18071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:15:21.625636 18035 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f2f1a94de04c28ab35e8d5e2ce038a. Latest consensus state: current_term: 1 leader_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } } }
I20260923 18:15:21.625798 18035 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:21.625990 18037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f2f1a94de04c28ab35e8d5e2ce038a. Latest consensus state: current_term: 1 leader_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } } }
I20260923 18:15:21.626112 18037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:21.638775 18082 mvcc.cc:205] Tried to move back new op lower bound from 7332607269346144256 to 7332607269260210176. Current Snapshot: MvccSnapshot[applied={T|T < 7332607269346144256}]
I20260923 18:15:21.639400 18034 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } } }
I20260923 18:15:21.639551 18034 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:21.641484 18035 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } } }
I20260923 18:15:21.641618 18035 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:21.642239 18085 mvcc.cc:205] Tried to move back new op lower bound from 7332607269346144256 to 7332607269260210176. Current Snapshot: MvccSnapshot[applied={T|T < 7332607269346144256}]
I20260923 18:15:21.643496 18071 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } } }
I20260923 18:15:21.643639 18071 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:21.644806 18037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f2f1a94de04c28ab35e8d5e2ce038a" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 37575 } } peers { permanent_uuid: "fa1189ae2b0c40eba3886eff1a595168" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 46207 } } peers { permanent_uuid: "12b95af4521a40cf961adbed04833b09" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 43121 } } }
I20260923 18:15:21.644942 18037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:21.648027 18086 mvcc.cc:205] Tried to move back new op lower bound from 7332607269346144256 to 7332607269260210176. Current Snapshot: MvccSnapshot[applied={T|T < 7332607269346144256}]
I20260923 18:15:21.649874 18078 catalog_manager.cc:1347] Generated new cluster ID: 161a040933ce44a2aecb659368470c5a
I20260923 18:15:21.649962 18078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:21.705096 18078 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:21.710579 18078 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:21.728890 18078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6f2f1a94de04c28ab35e8d5e2ce038a: Generated new TSK 0
I20260923 18:15:21.729888 18078 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:15:21.771960 17874 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:21.768389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54236 (local address 127.27.75.254:37575)
0923 18:15:21.768523 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.768528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.770427 (+ 1899us) server_negotiation.cc:407] Connection header received
0923 18:15:21.770929 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.770933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.770990 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.771043 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.771794 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54236: BlockingRecv error: recv got EOF from 127.0.0.1:54236 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260923 18:15:21.773450 17953 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:21.769326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.27.75.253:46207)
0923 18:15:21.769444 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.769449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.772469 (+ 3020us) server_negotiation.cc:407] Connection header received
0923 18:15:21.772637 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.772640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.772714 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.772765 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.773314 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43650: BlockingRecv error: recv got EOF from 127.0.0.1:43650 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 18:15:21.775086 18030 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:21.770138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49368 (local address 127.27.75.252:43121)
0923 18:15:21.770276 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.770281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.773849 (+ 3568us) server_negotiation.cc:407] Connection header received
0923 18:15:21.774128 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.774132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.774188 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.774244 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.774965 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49368: BlockingRecv error: recv got EOF from 127.0.0.1:49368 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260923 18:15:21.779294 17953 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:21.777637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.27.75.253:46207)
0923 18:15:21.777758 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.777762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.778261 (+ 499us) server_negotiation.cc:407] Connection header received
0923 18:15:21.778488 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.778491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.778568 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.778618 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.779177 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43658: BlockingRecv error: recv got EOF from 127.0.0.1:43658 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 18:15:21.785573 17953 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:21.783792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43662 (local address 127.27.75.253:46207)
0923 18:15:21.783916 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.783920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.784479 (+ 559us) server_negotiation.cc:407] Connection header received
0923 18:15:21.784667 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.784671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.784724 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.784772 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.785447 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43662: BlockingRecv error: recv got EOF from 127.0.0.1:43662 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260923 18:15:21.792086 17953 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:21.790375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.27.75.253:46207)
0923 18:15:21.790503 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.790507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.791096 (+ 589us) server_negotiation.cc:407] Connection header received
0923 18:15:21.791300 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.791304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.791360 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.791411 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.791971 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43672: BlockingRecv error: recv got EOF from 127.0.0.1:43672 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 18:15:21.799741 17953 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:21.798471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.27.75.253:46207)
0923 18:15:21.798621 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.798625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.798856 (+ 231us) server_negotiation.cc:407] Connection header received
0923 18:15:21.798985 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.798989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.799043 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.799108 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.799604 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43688: BlockingRecv error: recv got EOF from 127.0.0.1:43688 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260923 18:15:21.811100 17953 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:21.809883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.27.75.253:46207)
0923 18:15:21.810025 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.810030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.810276 (+ 246us) server_negotiation.cc:407] Connection header received
0923 18:15:21.810413 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.810417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.810468 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.810534 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.810980 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43702: BlockingRecv error: recv got EOF from 127.0.0.1:43702 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 18:15:21.830459 17953 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:21.829298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.27.75.253:46207)
0923 18:15:21.829472 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.829477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.829545 (+ 68us) server_negotiation.cc:407] Connection header received
0923 18:15:21.829673 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.829677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.829740 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.829843 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.830324 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43714: BlockingRecv error: recv got EOF from 127.0.0.1:43714 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260923 18:15:21.865883 17953 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:21.864633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.27.75.253:46207)
0923 18:15:21.864820 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.864824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.864952 (+ 128us) server_negotiation.cc:407] Connection header received
0923 18:15:21.865038 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.865041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.865092 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.865200 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.865732 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43728: BlockingRecv error: recv got EOF from 127.0.0.1:43728 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260923 18:15:21.933020 17953 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:21.931957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.27.75.253:46207)
0923 18:15:21.932123 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:21.932127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:21.932231 (+ 104us) server_negotiation.cc:407] Connection header received
0923 18:15:21.932354 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:21.932357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:21.932409 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:21.932482 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:21.932886 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43744: BlockingRecv error: recv got EOF from 127.0.0.1:43744 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 18:15:22.064435 17953 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:22.063254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.27.75.253:46207)
0923 18:15:22.063435 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:22.063440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:22.063540 (+ 100us) server_negotiation.cc:407] Connection header received
0923 18:15:22.063692 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:22.063695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:22.063745 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:22.063816 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:22.064307 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43750: BlockingRecv error: recv got EOF from 127.0.0.1:43750 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260923 18:15:22.323772 17953 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:22.322757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.27.75.253:46207)
0923 18:15:22.322956 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:22.322960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:22.322977 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:15:22.323082 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:22.323085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:22.323132 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:22.323199 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:22.323647 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43760: BlockingRecv error: recv got EOF from 127.0.0.1:43760 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20260923 18:15:22.604724 18059 catalog_manager.cc:1259] Loaded cluster ID: 161a040933ce44a2aecb659368470c5a
I20260923 18:15:22.604811 18059 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168: loading cluster ID for follower catalog manager: success
I20260923 18:15:22.606509 18059 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168: acquiring CA information for follower catalog manager: success
I20260923 18:15:22.607735 18059 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fa1189ae2b0c40eba3886eff1a595168: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:15:22.621742 18069 catalog_manager.cc:1259] Loaded cluster ID: 161a040933ce44a2aecb659368470c5a
I20260923 18:15:22.621835 18069 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09: loading cluster ID for follower catalog manager: success
I20260923 18:15:22.623602 18069 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09: acquiring CA information for follower catalog manager: success
I20260923 18:15:22.624861 18069 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12b95af4521a40cf961adbed04833b09: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:15:22.841945 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:22.837844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.27.75.253:46207)
0923 18:15:22.838088 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:22.838092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:22.838108 (+ 16us) server_negotiation.cc:407] Connection header received
0923 18:15:22.838244 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:22.838247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:22.838371 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:22.838467 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:15:22.838901 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:22.839440 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:22.840338 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:22.840571 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:22.840653 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:22.840992 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:15:22.841484 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:15:22.841579 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:22.841662 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 18:15:22.841729 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":64,"threads_started":1}
I20260923 18:15:22.849440 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:22.845396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.27.75.252:43121)
0923 18:15:22.845621 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:22.845624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:22.845638 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:15:22.845719 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:22.845722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:22.845867 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:22.845976 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:15:22.846253 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:22.846796 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:22.847697 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:22.847992 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:22.848080 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:22.848512 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:15:22.849014 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:15:22.849139 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:22.849251 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 18:15:22.849311 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260923 18:15:22.856684 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:22.852875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.27.75.253:46207)
0923 18:15:22.853057 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:22.853061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:22.853079 (+ 18us) server_negotiation.cc:407] Connection header received
0923 18:15:22.853254 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:22.853258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:22.853311 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:22.853360 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:15:22.853711 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:22.854403 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:22.855488 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:22.855813 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:22.855911 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:22.856165 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:15:22.856273 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:15:22.856318 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:22.856480 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 18:15:22.856543 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 18:15:22.857059 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:22.853430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.27.75.252:43121)
0923 18:15:22.853537 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:22.853541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:22.853554 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:15:22.853650 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:22.853653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:22.853703 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:22.853782 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:15:22.854009 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:22.854800 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:22.855855 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:22.856166 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:22.856251 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:22.856577 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:15:22.856661 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:15:22.856702 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:22.856874 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 18:15:22.856928 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:15:22.857584 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:22.852808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.27.75.254:37575)
0923 18:15:22.853265 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:22.853270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:22.853285 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:15:22.853367 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:22.853370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:22.853531 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:22.853912 (+ 381us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:15:22.854011 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:22.854822 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:22.855805 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:22.856125 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:22.856456 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:22.856599 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:15:22.856971 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:15:22.857092 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:22.857377 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 18:15:22.857444 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":80,"threads_started":1}
I20260923 18:15:22.868815 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 17802
I20260923 18:15:22.874819 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 17878
I20260923 18:15:22.879791 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 17954
2026-09-23T18:15:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3055 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 18:15:22 dist-test-slave-wsx5 krb5kdc[18174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 18:15:22 dist-test-slave-wsx5 krb5kdc[18174](info): set up 2 sockets
Sep 23 18:15:22 dist-test-slave-wsx5 krb5kdc[18174](info): commencing operation
krb5kdc: starting...
W20260923 18:15:23.847615 27951 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/test-user.keytab.
Sep 23 18:15:23 dist-test-slave-wsx5 krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T18:15:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T18:15:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
Entry for principal kudu/127.27.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab.
I20260923 18:15:23.972498 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:41241
--webserver_interface=127.27.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:41241,127.27.75.253:41381,127.27.75.252:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 18:15:24.084623 18187 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:15:24.084944 18187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:24.084967 18187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:24.085005 18187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:24.089075 18187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:24.089166 18187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:24.089187 18187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:24.089205 18187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:24.089218 18187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:24.094659 18187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:41241,127.27.75.253:41381,127.27.75.252:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.254.keytab
--principal=kudu/127.27.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.254:41241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:24.095750 18187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:24.096729 18187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:24.105250 18193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:24.105240 18192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:24.105244 18195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:24.105284 18187 server_base.cc:1034] running on GCE node
I20260923 18:15:24.106125 18187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:24.107288 18187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:24.108472 18187 hybrid_clock.cc:648] HybridClock initialized: now 1790187324108454 us; error 33 us; skew 500 ppm
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:24.111217 18187 init.cc:377] Logged in from keytab as kudu/127.27.75.254@KRBTEST.COM (short username kudu)
I20260923 18:15:24.112133 18187 webserver.cc:460] Webserver started at http://127.27.75.254:36007/ using document root <none> and password file <none>
I20260923 18:15:24.112686 18187 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:24.112753 18187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:24.112962 18187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:24.115074 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/data/instance:
uuid: "484444f270d74131b2e7ef1d270409e4"
format_stamp: "Formatted at 2026-09-23 18:15:24 on dist-test-slave-wsx5"
server_key: "865f82dc70190e1c00e99edbed0a6b3c"
server_key_iv: "7872fcb462e0c373852838bbf7d12ded"
server_key_version: "encryptionkey@0"
I20260923 18:15:24.115648 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/wal/instance:
uuid: "484444f270d74131b2e7ef1d270409e4"
format_stamp: "Formatted at 2026-09-23 18:15:24 on dist-test-slave-wsx5"
server_key: "865f82dc70190e1c00e99edbed0a6b3c"
server_key_iv: "7872fcb462e0c373852838bbf7d12ded"
server_key_version: "encryptionkey@0"
I20260923 18:15:24.119540 18187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 18:15:24.122061 18202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:24.123353 18187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 18:15:24.123476 18187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/wal
uuid: "484444f270d74131b2e7ef1d270409e4"
format_stamp: "Formatted at 2026-09-23 18:15:24 on dist-test-slave-wsx5"
server_key: "865f82dc70190e1c00e99edbed0a6b3c"
server_key_iv: "7872fcb462e0c373852838bbf7d12ded"
server_key_version: "encryptionkey@0"
I20260923 18:15:24.123582 18187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:24.147907 18187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:24.151475 18187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:24.151676 18187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:24.159132 18187 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.254:41241
I20260923 18:15:24.159412 18254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.254:41241 every 8 connection(s)
I20260923 18:15:24.160175 18187 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/data/info.pb
I20260923 18:15:24.163228 18255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:24.167129 18255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:24.169287 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 18187
I20260923 18:15:24.169428 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-0/wal/instance
I20260923 18:15:24.169714 27951 external_mini_cluster.cc:1428] Setting key ac75a8f65a3324362ac3b4f1c7204116
WARNING: no policy specified for kudu/127.27.75.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.253@KRBTEST.COM" created.
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:15:24.187070 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.169554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:41241 (local address 127.0.0.1:58962)
0923 18:15:24.170132 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.170189 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:24.171337 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:24.172448 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:24.172453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:24.172866 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:24.173172 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.173192 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.174864 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.174867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.175643 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.175652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.176689 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.176692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.176842 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.177457 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:24.177475 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:24.181396 (+ 3921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:24.183776 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.183788 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.183800 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.184094 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.184456 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.184467 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.184469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.184629 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.185349 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:24.185356 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:24.185541 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 18:15:24.185744 (+ 203us) client_negotiation.cc:240] Negotiation successful
0923 18:15:24.186240 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":112,"thread_start_us":152,"threads_started":1}
I20260923 18:15:24.187069 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.169554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.27.75.254:41241)
0923 18:15:24.170136 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.170152 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.170306 (+ 154us) server_negotiation.cc:407] Connection header received
0923 18:15:24.171903 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.171910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.172188 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.172403 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.173337 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.174699 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.175804 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.176412 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.176629 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.181589 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.181595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.181608 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.181634 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.183647 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.184230 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.184235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.184237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.184367 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.184741 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.184744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.184746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.185226 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.185337 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.185865 (+ 528us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.186151 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":456,"spinlock_wait_cycles":30080,"thread_start_us":162,"threads_started":1}
I20260923 18:15:24.189618 18255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:24.190415 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.190134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:41381 (local address 127.0.0.1:43102)
0923 18:15:24.190252 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.190346 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 18:15:24.190609 18204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.75.253:41381: connect: Connection refused (error 111)
I20260923 18:15:24.191524 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.191293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:41381 (local address 127.0.0.1:43106)
0923 18:15:24.191405 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.191460 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 18:15:24.191686 18255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.253:41381: Network error: Client connection negotiation failed: client connection to 127.27.75.253:41381: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab.
Entry for principal kudu/127.27.75.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab.
I20260923 18:15:24.198455 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.253:41381
--webserver_interface=127.27.75.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab
--principal=kudu/127.27.75.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:41241,127.27.75.253:41381,127.27.75.252:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:15:24.240979 18255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } attempt: 1
I20260923 18:15:24.241982 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.241641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:41381 (local address 127.0.0.1:43110)
0923 18:15:24.241811 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.241890 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 18:15:24.242724 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.242508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:41381 (local address 127.0.0.1:43122)
0923 18:15:24.242606 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.242656 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 18:15:24.242856 18255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.253:41381: Network error: Client connection negotiation failed: client connection to 127.27.75.253:41381: connect: Connection refused (error 111)
I20260923 18:15:24.311070 18255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } attempt: 2
I20260923 18:15:24.311959 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.311628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:41381 (local address 127.0.0.1:43126)
0923 18:15:24.311796 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.311867 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 18:15:24.312762 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.312530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:41381 (local address 127.0.0.1:43132)
0923 18:15:24.312627 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.312687 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.253:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 18:15:24.312903 18255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.253:41381: Network error: Client connection negotiation failed: client connection to 127.27.75.253:41381: connect: Connection refused (error 111)
W20260923 18:15:24.313216 18262 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:15:24.313513 18262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:24.313555 18262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:24.313619 18262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:24.317821 18262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:24.317890 18262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:24.317909 18262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:24.317925 18262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:24.317939 18262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:24.323556 18262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:41241,127.27.75.253:41381,127.27.75.252:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.253.keytab
--principal=kudu/127.27.75.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.253:41381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.75.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:24.324702 18262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:24.325784 18262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:24.334769 18267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:24.334770 18268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:24.334769 18270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:24.335028 18262 server_base.cc:1034] running on GCE node
I20260923 18:15:24.335626 18262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:24.336853 18262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:24.338063 18262 hybrid_clock.cc:648] HybridClock initialized: now 1790187324338026 us; error 52 us; skew 500 ppm
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:24.341144 18262 init.cc:377] Logged in from keytab as kudu/127.27.75.253@KRBTEST.COM (short username kudu)
I20260923 18:15:24.342203 18262 webserver.cc:460] Webserver started at http://127.27.75.253:42153/ using document root <none> and password file <none>
I20260923 18:15:24.342793 18262 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:24.342882 18262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:24.343110 18262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:24.345438 18262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/data/instance:
uuid: "946b11efc5be47a191e3a5f1303831dd"
format_stamp: "Formatted at 2026-09-23 18:15:24 on dist-test-slave-wsx5"
server_key: "ba4cf569382ff9b974867eb87181f144"
server_key_iv: "6af90a27ad8a81de4965a5b831bf72c6"
server_key_version: "encryptionkey@0"
I20260923 18:15:24.346027 18262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/wal/instance:
uuid: "946b11efc5be47a191e3a5f1303831dd"
format_stamp: "Formatted at 2026-09-23 18:15:24 on dist-test-slave-wsx5"
server_key: "ba4cf569382ff9b974867eb87181f144"
server_key_iv: "6af90a27ad8a81de4965a5b831bf72c6"
server_key_version: "encryptionkey@0"
I20260923 18:15:24.349850 18262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 18:15:24.352463 18278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:24.353940 18262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:15:24.354065 18262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/wal
uuid: "946b11efc5be47a191e3a5f1303831dd"
format_stamp: "Formatted at 2026-09-23 18:15:24 on dist-test-slave-wsx5"
server_key: "ba4cf569382ff9b974867eb87181f144"
server_key_iv: "6af90a27ad8a81de4965a5b831bf72c6"
server_key_version: "encryptionkey@0"
I20260923 18:15:24.354172 18262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:24.377027 18262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:24.380236 18262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:24.380417 18262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:24.386937 18262 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.253:41381
I20260923 18:15:24.387192 18330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.253:41381 every 8 connection(s)
I20260923 18:15:24.387900 18262 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/data/info.pb
I20260923 18:15:24.390753 18331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 18:15:24.394619 18331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:24.395212 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 18262
I20260923 18:15:24.395360 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-1/wal/instance
I20260923 18:15:24.395736 27951 external_mini_cluster.cc:1428] Setting key 9066df431205d3935eac54925babdb6e
WARNING: no policy specified for kudu/127.27.75.252@KRBTEST.COM; defaulting to no policy
I20260923 18:15:24.404071 18255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.75.252@KRBTEST.COM" created.
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
I20260923 18:15:24.413003 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.396931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.27.75.254:41241)
0923 18:15:24.397088 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.397093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.398174 (+ 1081us) server_negotiation.cc:407] Connection header received
0923 18:15:24.399393 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.399396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.399450 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.399530 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.401227 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.401855 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.403618 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.403930 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.404054 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.408288 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.408291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.408294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.408327 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.409908 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.410483 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.410486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.410488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.410565 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.410926 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.410929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.410931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.411162 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.411256 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.412705 (+ 1449us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.412850 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 18:15:24.413704 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.397417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:41241 (local address 127.0.0.1:58968)
0923 18:15:24.397973 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.398033 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:24.399121 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:24.399742 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:24.399753 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:24.400303 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:24.401009 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.401031 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.402001 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.402005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.403446 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.403460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.404123 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.404129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.404374 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.404963 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:24.404984 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:24.408132 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:24.410022 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.410029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.410049 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.410374 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.410659 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.410662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.410664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.410822 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.411322 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:24.411329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:24.411656 (+ 327us) client_negotiation.cc:769] Sending connection context
0923 18:15:24.411912 (+ 256us) client_negotiation.cc:240] Negotiation successful
0923 18:15:24.412322 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":126,"threads_started":1}
I20260923 18:15:24.414803 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.404551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:41381 (local address 127.0.0.1:43144)
0923 18:15:24.404947 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.404964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:24.405045 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:24.405543 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:24.405545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:24.405726 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:24.405831 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.405837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.406851 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.406855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.407445 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.407456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.408409 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.408412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.408509 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.409071 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:24.409087 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:24.410555 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:24.412901 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.412905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.412907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.413211 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.413550 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.413553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.413555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.413651 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.414255 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:24.414259 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:24.414365 (+ 106us) client_negotiation.cc:769] Sending connection context
0923 18:15:24.414473 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 18:15:24.414628 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 18:15:24.415385 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.404722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43144 (local address 127.27.75.253:41381)
0923 18:15:24.405099 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.405112 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.405171 (+ 59us) server_negotiation.cc:407] Connection header received
0923 18:15:24.405209 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.405214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.405387 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.405541 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.405947 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.406706 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.407584 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.408132 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.408247 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.410714 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.410725 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.410741 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.410779 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.412748 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.413344 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.413350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.413353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.413444 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.413752 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.413755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.413757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.414072 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.414204 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.415051 (+ 847us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.415213 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":123,"threads_started":1}
I20260923 18:15:24.415702 18331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:24.418046 18255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:24.418828 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.418571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59474)
0923 18:15:24.418705 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.418759 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 18:15:24.419693 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.419415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59486)
0923 18:15:24.419573 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.419626 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260923 18:15:24.419847 18255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:41481: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.75.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab.
Entry for principal kudu/127.27.75.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab.
I20260923 18:15:24.424451 27951 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
/tmp/dist-test-tasknO4ss8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.252:41481
--webserver_interface=127.27.75.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab
--principal=kudu/127.27.75.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.75.212:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.75.254:41241,127.27.75.253:41381,127.27.75.252:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 18:15:24.424954 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.416281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:41381 (local address 127.0.0.1:43154)
0923 18:15:24.416387 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.416405 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:24.416480 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:24.417084 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:24.417086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:24.417279 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:24.417378 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.417383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.418300 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.418304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.419061 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.419072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.419619 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.419622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.419794 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.420386 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:24.420398 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:24.421711 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:24.423481 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.423485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.423487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.423720 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.424019 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.424022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.424024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.424120 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.424441 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:24.424444 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:24.424549 (+ 105us) client_negotiation.cc:769] Sending connection context
0923 18:15:24.424654 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 18:15:24.424795 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 18:15:24.425420 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.416755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43154 (local address 127.27.75.253:41381)
0923 18:15:24.416886 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.416890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.416904 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:15:24.416936 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.416939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.416979 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.417060 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.417480 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.418132 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.419166 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.419494 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.419593 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.421880 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.421883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.421885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.421907 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.423358 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.423865 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.423867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.423869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.423940 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.424197 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.424199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.424200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.424360 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.424425 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.425186 (+ 761us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.425295 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 18:15:24.426129 18331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:24.426826 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.426558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59502)
0923 18:15:24.426673 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.426757 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 18:15:24.427385 18280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
I20260923 18:15:24.428298 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.428095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59514)
0923 18:15:24.428195 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.428241 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 18:15:24.428453 18331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:41481: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
I20260923 18:15:24.451015 18255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } attempt: 1
I20260923 18:15:24.451972 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.451628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59516)
0923 18:15:24.451824 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.451890 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 18:15:24.452733 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.452465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59532)
0923 18:15:24.452601 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.452664 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 18:15:24.452869 18255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:41481: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
I20260923 18:15:24.477754 18331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } attempt: 1
I20260923 18:15:24.478708 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.478374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59542)
0923 18:15:24.478553 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.478621 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 18:15:24.479533 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.479310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59548)
0923 18:15:24.479426 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.479471 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 18:15:24.479681 18331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:41481: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
I20260923 18:15:24.528029 18255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } attempt: 2
I20260923 18:15:24.528839 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.528558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59552)
0923 18:15:24.528698 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.528761 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 18:15:24.529610 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.529387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59556)
0923 18:15:24.529495 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.529545 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 18:15:24.529738 18255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:41481: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
W20260923 18:15:24.538393 18337 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 18:15:24.538756 18337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 18:15:24.538792 18337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 18:15:24.538861 18337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 18:15:24.543377 18337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 18:15:24.543470 18337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 18:15:24.543502 18337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 18:15:24.543524 18337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 18:15:24.543545 18337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 18:15:24.547863 18331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } attempt: 2
I20260923 18:15:24.548782 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.548428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59570)
0923 18:15:24.548638 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.548703 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 18:15:24.549512 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.549296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59582)
0923 18:15:24.549391 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.549446 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 18:15:24.549635 18331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.75.252:41481: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
I20260923 18:15:24.549690 18337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.75.212:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.75.254:41241,127.27.75.253:41381,127.27.75.252:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/krb5kdc/kudu_127.27.75.252.keytab
--principal=kudu/127.27.75.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.75.252:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.75.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 17:12:33 UTC on ccycloud.quasar-bzzyns.root.comops.site
build id 4667750
I20260923 18:15:24.550722 18337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 18:15:24.551661 18337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 18:15:24.560158 18346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:24.560158 18343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 18:15:24.560160 18344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 18:15:24.560469 18337 server_base.cc:1034] running on GCE node
I20260923 18:15:24.561111 18337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 18:15:24.562332 18337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 18:15:24.563529 18337 hybrid_clock.cc:648] HybridClock initialized: now 1790187324563507 us; error 34 us; skew 500 ppm
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 18:15:24.566514 18337 init.cc:377] Logged in from keytab as kudu/127.27.75.252@KRBTEST.COM (short username kudu)
I20260923 18:15:24.567456 18337 webserver.cc:460] Webserver started at http://127.27.75.252:41881/ using document root <none> and password file <none>
I20260923 18:15:24.567998 18337 fs_manager.cc:359] Metadata directory not provided
I20260923 18:15:24.568058 18337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 18:15:24.568226 18337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 18:15:24.570408 18337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/data/instance:
uuid: "58279e179f424e429fb72bfaba7d044e"
format_stamp: "Formatted at 2026-09-23 18:15:24 on dist-test-slave-wsx5"
server_key: "2cd907a3b14d44df66f81ae6e715777c"
server_key_iv: "ca569de87c68869dd585cc7ab750ab0f"
server_key_version: "encryptionkey@0"
I20260923 18:15:24.570992 18337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/wal/instance:
uuid: "58279e179f424e429fb72bfaba7d044e"
format_stamp: "Formatted at 2026-09-23 18:15:24 on dist-test-slave-wsx5"
server_key: "2cd907a3b14d44df66f81ae6e715777c"
server_key_iv: "ca569de87c68869dd585cc7ab750ab0f"
server_key_version: "encryptionkey@0"
I20260923 18:15:24.575081 18337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 18:15:24.577697 18353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 18:15:24.579106 18337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 18:15:24.579228 18337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/data,/tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/wal
uuid: "58279e179f424e429fb72bfaba7d044e"
format_stamp: "Formatted at 2026-09-23 18:15:24 on dist-test-slave-wsx5"
server_key: "2cd907a3b14d44df66f81ae6e715777c"
server_key_iv: "ca569de87c68869dd585cc7ab750ab0f"
server_key_version: "encryptionkey@0"
I20260923 18:15:24.579358 18337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 18:15:24.619201 18337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 18:15:24.622462 18337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 18:15:24.622637 18337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 18:15:24.628914 18255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } attempt: 3
I20260923 18:15:24.629774 18337 rpc_server.cc:307] RPC server started. Bound to: 127.27.75.252:41481
I20260923 18:15:24.629859 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.629447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59598)
0923 18:15:24.629620 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.629769 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.75.252:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 18:15:24.630048 18406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.75.252:41481 every 8 connection(s)
I20260923 18:15:24.631033 18337 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/data/info.pb
I20260923 18:15:24.631868 27951 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu as pid 18337
I20260923 18:15:24.632004 27951 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknO4ss8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790187042487232-27951-0/minicluster-data/master-2/wal/instance
I20260923 18:15:24.632295 27951 external_mini_cluster.cc:1428] Setting key 06f32d899b676ef54cd230cccd3f5d56
I20260923 18:15:24.634778 18407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.254@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
I20260923 18:15:24.639362 18407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
I20260923 18:15:24.640784 18331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } attempt: 3
I20260923 18:15:24.645172 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.630432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59612)
0923 18:15:24.630565 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.630581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:24.630682 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:24.633671 (+ 2989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:24.633674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:24.633908 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:24.634041 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.634050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.635738 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.635742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.636346 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.636358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.637407 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.637410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.637497 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.638012 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:24.638028 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:24.639567 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:24.643150 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.643153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.643155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.643388 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.643740 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.643743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.643745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.643830 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.644660 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:24.644663 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:24.644767 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 18:15:24.644853 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 18:15:24.644982 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.253@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
I20260923 18:15:24.647131 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.631268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.27.75.252:41481)
0923 18:15:24.631844 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.631864 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.631908 (+ 44us) server_negotiation.cc:407] Connection header received
0923 18:15:24.632800 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.632824 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.633334 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.633707 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.634194 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.635610 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.636484 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.637256 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.637402 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.639852 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.639859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.639876 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.639910 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.643051 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.643492 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.643495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.643496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.643620 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.643936 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.643939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.643941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.644540 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.645604 (+ 1064us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.645782 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.646105 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":355,"spinlock_wait_cycles":9216,"thread_start_us":138,"threads_started":1}
I20260923 18:15:24.647131 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.633176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.27.75.252:41481)
0923 18:15:24.634230 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.634239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.634250 (+ 11us) server_negotiation.cc:407] Connection header received
0923 18:15:24.634282 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.634285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.634323 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.634407 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.635298 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.636062 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.637237 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.637559 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.637663 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.640638 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.640642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.640644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.640671 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.642952 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.643451 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.643457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.643464 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.643629 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.643936 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.643939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.643941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.644540 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.646239 (+ 1699us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.646309 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.646410 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":991,"spinlock_wait_cycles":21120,"thread_start_us":113,"threads_started":1}
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:15:24.651676 18255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4: Bootstrap starting.
I20260923 18:15:24.652290 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.642179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.27.75.254:41241)
0923 18:15:24.642301 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.642305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.642717 (+ 412us) server_negotiation.cc:407] Connection header received
0923 18:15:24.642903 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.642906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.642956 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.643030 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.643902 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.644619 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.645910 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.646230 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.646318 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.648846 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.648850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.648853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.648882 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.650556 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.651047 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.651050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.651052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.651131 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.651389 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.651392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.651394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.651574 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.651658 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.652008 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.652130 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 18:15:24.652309 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.641396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.254:41241 (local address 127.0.0.1:58970)
0923 18:15:24.642587 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.642638 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:24.642810 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:24.643155 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:24.643161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:24.643608 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:24.643771 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.643778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.644736 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.644738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.645774 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.645786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.646463 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.646466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.646583 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.647048 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:24.647062 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:24.648676 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:24.650665 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.650674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.650680 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.650955 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.651216 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.651219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.651221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.651307 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.651670 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:24.651676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:24.651806 (+ 130us) client_negotiation.cc:769] Sending connection context
0923 18:15:24.651942 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 18:15:24.652132 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":75,"threads_started":1}
I20260923 18:15:24.653419 18407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:24.654215 18255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:24.654938 18255 log.cc:824] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:24.656883 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.641272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59628)
0923 18:15:24.641472 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.641486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:24.641566 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:24.642054 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:24.642057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:24.642234 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:24.642352 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.642360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.643403 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.643408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.644184 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.644194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.644748 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.644751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.644844 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.645353 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:24.645368 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:24.647048 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:24.649207 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.649211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.649213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.649462 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.655878 (+ 6416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.655882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.655884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.655990 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.656405 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:24.656408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:24.656508 (+ 100us) client_negotiation.cc:769] Sending connection context
0923 18:15:24.656584 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 18:15:24.656720 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260923 18:15:24.657244 18255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4: No bootstrap required, opened a new log
I20260923 18:15:24.657938 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.641275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.27.75.252:41481)
0923 18:15:24.641816 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.641824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.641837 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:15:24.641869 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.641871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.641912 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.642451 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.642546 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.643279 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.644316 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.644630 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.647176 (+ 2546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.647507 (+ 331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.647511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.647514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.647535 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.649096 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.655622 (+ 6526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.655627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.655629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.655757 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.656096 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.656099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.656101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.656308 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.657481 (+ 1173us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.657614 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.657770 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"spinlock_wait_cycles":9088,"thread_start_us":328,"threads_started":1}
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.254@KRBTEST.COM
I20260923 18:15:24.661530 18331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd: Bootstrap starting.
I20260923 18:15:24.661600 18255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } }
I20260923 18:15:24.661854 18255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484444f270d74131b2e7ef1d270409e4, State: Initialized, Role: FOLLOWER
I20260923 18:15:24.662200 18255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } }
I20260923 18:15:24.663367 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.653985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.27.75.253:41381)
0923 18:15:24.654094 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.654099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.654153 (+ 54us) server_negotiation.cc:407] Connection header received
0923 18:15:24.654280 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.654284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.654332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.654400 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.654787 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.655339 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.656374 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.656715 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.656782 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.659217 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.659221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.659223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.659513 (+ 290us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.661025 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.661555 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.661558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.661560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.661641 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.661948 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.661951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.661953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.662419 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.662491 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.663101 (+ 610us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.663220 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 18:15:24.663681 18255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:24.663857 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.653860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.253:41381 (local address 127.0.0.1:43164)
0923 18:15:24.654099 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.654112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:24.654195 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:24.654414 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:24.654416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:24.654576 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:24.654673 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.654678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.655455 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.655458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.656246 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.656255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.656928 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.656931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.657025 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.657533 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:24.657547 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:24.659068 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:24.661203 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.661206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.661208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.661443 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.661764 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.661767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.661769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.661853 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.662563 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:24.662566 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:24.662903 (+ 337us) client_negotiation.cc:769] Sending connection context
0923 18:15:24.663552 (+ 649us) client_negotiation.cc:240] Negotiation successful
0923 18:15:24.663708 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260923 18:15:24.664374 18416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } } }
I20260923 18:15:24.664511 18416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:24.665266 18407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } has no permanent_uuid. Determining permanent_uuid...
I20260923 18:15:24.666123 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.649952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.27.75.254:41241)
0923 18:15:24.650797 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.650809 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.655492 (+ 4683us) server_negotiation.cc:407] Connection header received
0923 18:15:24.655626 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.655629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.655677 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.656512 (+ 835us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.656610 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.657393 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.658480 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.658861 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.659303 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.661882 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.661886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.661888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.661917 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.663485 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.664148 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.664151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.664152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.664254 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.664599 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.664602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.664605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.664836 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.665722 (+ 886us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.665806 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.665932 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":92,"threads_started":1}
I20260923 18:15:24.666975 18331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd: Neither blocks nor log segments found. Creating new log.
W20260923 18:15:24.669669 18427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:15:24.669760 18427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:15:24.669157 18187 master_runner.cc:186] Error getting master registration for 127.27.75.253:41381: OK, instance_id { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" instance_seqno: 1790187324385172 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 18:15:24.672145 18187 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:15:24.672492 18331 log.cc:824] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd: Log is configured to *not* fsync() on all Append() calls
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.75.253@KRBTEST.COM
I20260923 18:15:24.675032 18331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd: No bootstrap required, opened a new log
Sep 23 18:15:24 dist-test-slave-wsx5 krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790187324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.75.252@KRBTEST.COM for kudu/127.27.75.252@KRBTEST.COM
I20260923 18:15:24.678642 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.667527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.27.75.253:41381)
0923 18:15:24.668268 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.668272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.668284 (+ 12us) server_negotiation.cc:407] Connection header received
0923 18:15:24.668318 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.668321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.668366 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.668434 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.669359 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.670109 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.671203 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.671551 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.671653 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.674818 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.674823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.674826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.674855 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.676523 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.677038 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.677042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.677044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.677169 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.677506 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.677509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.677511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.677734 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.677896 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.678347 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.678476 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658}
I20260923 18:15:24.681506 18331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } }
I20260923 18:15:24.681838 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.665739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.75.252:41481 (local address 127.0.0.1:59640)
0923 18:15:24.665991 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 18:15:24.666005 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 18:15:24.666082 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 18:15:24.673381 (+ 7299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 18:15:24.673384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 18:15:24.673567 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0923 18:15:24.673677 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.673685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.674653 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.674656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.675375 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 18:15:24.675388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.675931 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.675935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 18:15:24.676017 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.676400 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 18:15:24.676412 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 18:15:24.677926 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 18:15:24.680016 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.680019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.680021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.680246 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.680913 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 18:15:24.680916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 18:15:24.680918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 18:15:24.680999 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 18:15:24.681410 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 18:15:24.681412 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 18:15:24.681485 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 18:15:24.681558 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 18:15:24.681684 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260923 18:15:24.681766 18331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946b11efc5be47a191e3a5f1303831dd, State: Initialized, Role: FOLLOWER
I20260923 18:15:24.682337 18331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } }
I20260923 18:15:24.682739 27951 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 18:15:24.682852 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.665981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.27.75.252:41481)
0923 18:15:24.673074 (+ 7093us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.673079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.673096 (+ 17us) server_negotiation.cc:407] Connection header received
0923 18:15:24.673180 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.673183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.673238 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.673781 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.673841 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.674543 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.675501 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.675814 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.676231 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.678062 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.678065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.678068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.678090 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.679915 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.680362 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.680365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.680367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.680819 (+ 452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.681085 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.681087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.681089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.681306 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.682112 (+ 806us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.682185 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.682693 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6985}
I20260923 18:15:24.683990 18331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:24.684228 18229 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "484444f270d74131b2e7ef1d270409e4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58974
I20260923 18:15:24.684489 18229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 18:15:24.684592 18229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:24.685876 18229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } }
I20260923 18:15:24.686014 18407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e: Bootstrap starting.
I20260923 18:15:24.686827 18229 leader_election.cc:288] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [CANDIDATE]: Term 1 election: Requested vote from peers 946b11efc5be47a191e3a5f1303831dd (127.27.75.253:41381), 58279e179f424e429fb72bfaba7d044e (127.27.75.252:41481)
I20260923 18:15:24.687577 18305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "484444f270d74131b2e7ef1d270409e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "946b11efc5be47a191e3a5f1303831dd"
I20260923 18:15:24.687811 18305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:24.688966 18407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e: Neither blocks nor log segments found. Creating new log.
I20260923 18:15:24.689308 18305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484444f270d74131b2e7ef1d270409e4 in term 1.
I20260923 18:15:24.689810 18204 leader_election.cc:302] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484444f270d74131b2e7ef1d270409e4, 946b11efc5be47a191e3a5f1303831dd; no voters:
I20260923 18:15:24.689975 18407 log.cc:824] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e: Log is configured to *not* fsync() on all Append() calls
I20260923 18:15:24.690127 18416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 18:15:24.690339 18416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [term 1 LEADER]: Becoming Leader. State: Replica: 484444f270d74131b2e7ef1d270409e4, State: Running, Role: LEADER
I20260923 18:15:24.690706 18416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } }
I20260923 18:15:24.692492 18407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e: No bootstrap required, opened a new log
I20260923 18:15:24.692700 18416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484444f270d74131b2e7ef1d270409e4. Latest consensus state: current_term: 1 leader_uuid: "484444f270d74131b2e7ef1d270409e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } } }
I20260923 18:15:24.692835 18416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [sys.catalog]: This master's current role is: LEADER
W20260923 18:15:24.694783 18441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:15:24.694880 18441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:15:24.697475 18407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } }
I20260923 18:15:24.697734 18407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58279e179f424e429fb72bfaba7d044e, State: Initialized, Role: FOLLOWER
I20260923 18:15:24.698071 18407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } }
I20260923 18:15:24.699501 18407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 18:15:24.696882 18262 master_runner.cc:186] Error getting master registration for 127.27.75.252:41481: OK, instance_id { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" instance_seqno: 1790187324627688 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 18:15:24.700716 18380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "484444f270d74131b2e7ef1d270409e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "58279e179f424e429fb72bfaba7d044e"
I20260923 18:15:24.700961 18380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [term 0 FOLLOWER]: Advancing to term 1
I20260923 18:15:24.702302 18380 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484444f270d74131b2e7ef1d270409e4 in term 1.
I20260923 18:15:24.702816 18262 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 18:15:24.701841 18430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } } }
I20260923 18:15:24.702999 18430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:24.702745 18451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } } }
I20260923 18:15:24.703161 18449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 18:15:24.703197 18451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:24.704495 18449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260923 18:15:24.705912 18462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 18:15:24.705986 18462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 18:15:24.715368 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.702117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.27.75.254:41241)
0923 18:15:24.702228 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.702232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.704222 (+ 1990us) server_negotiation.cc:407] Connection header received
0923 18:15:24.704408 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.704411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.704461 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.704508 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.707335 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.708038 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.709167 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.709543 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.709651 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.711526 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.711530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.711533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.711565 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.713416 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.713905 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.713909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.713911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.713999 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.714316 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.714320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.714322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.714559 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.714663 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.715037 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.715173 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 18:15:24.715438 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.701649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.27.75.253:41381)
0923 18:15:24.701787 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.701791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.705685 (+ 3894us) server_negotiation.cc:407] Connection header received
0923 18:15:24.705818 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.705821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.705871 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.705919 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.706824 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.707596 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.709991 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.710339 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.710437 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.712070 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.712075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.712078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.712108 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.713676 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.714132 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.714135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.714137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.714230 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.714506 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.714509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.714511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.714714 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.714820 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.715155 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.715285 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 18:15:24.725167 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:24.703804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.27.75.252:41481)
0923 18:15:24.703903 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.703907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.715937 (+ 12030us) server_negotiation.cc:407] Connection header received
0923 18:15:24.716117 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.716121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.716181 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.716233 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.717101 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.717871 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.718903 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:24.719202 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:24.719274 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:24.721020 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 18:15:24.721025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 18:15:24.721028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 18:15:24.721057 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 18:15:24.723137 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.723673 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.723676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.723678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.723772 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 18:15:24.724076 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 18:15:24.724080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 18:15:24.724082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 18:15:24.724328 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 18:15:24.724431 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:24.724803 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 18:15:24.724958 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 18:15:24.727077 18305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [term 1 FOLLOWER]: Refusing update from remote peer 484444f270d74131b2e7ef1d270409e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:15:24.727075 18380 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [term 1 FOLLOWER]: Refusing update from remote peer 484444f270d74131b2e7ef1d270409e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 18:15:24.727865 18416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:15:24.728319 18416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 18:15:24.735942 18451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484444f270d74131b2e7ef1d270409e4. Latest consensus state: current_term: 1 leader_uuid: "484444f270d74131b2e7ef1d270409e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } } }
I20260923 18:15:24.736127 18451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:24.737437 18430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484444f270d74131b2e7ef1d270409e4. Latest consensus state: current_term: 1 leader_uuid: "484444f270d74131b2e7ef1d270409e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } } }
I20260923 18:15:24.737572 18430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:24.748334 18446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "484444f270d74131b2e7ef1d270409e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } } }
I20260923 18:15:24.748515 18446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:24.748649 18467 mvcc.cc:205] Tried to move back new op lower bound from 7332607282051506176 to 7332607281936855040. Current Snapshot: MvccSnapshot[applied={T|T < 7332607282051506176}]
I20260923 18:15:24.750286 18469 mvcc.cc:205] Tried to move back new op lower bound from 7332607282051506176 to 7332607281936855040. Current Snapshot: MvccSnapshot[applied={T|T < 7332607282051506176}]
I20260923 18:15:24.750473 18451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "484444f270d74131b2e7ef1d270409e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } } }
I20260923 18:15:24.750595 18451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:24.754587 18416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "484444f270d74131b2e7ef1d270409e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } } }
I20260923 18:15:24.754745 18416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4 [sys.catalog]: This master's current role is: LEADER
I20260923 18:15:24.755090 18449 catalog_manager.cc:1347] Generated new cluster ID: 85eb06f525764e3d8434283babd0f32a
I20260923 18:15:24.755162 18449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 18:15:24.755940 18468 mvcc.cc:205] Tried to move back new op lower bound from 7332607282051506176 to 7332607281936855040. Current Snapshot: MvccSnapshot[applied={T|T < 7332607282051506176}]
I20260923 18:15:24.756114 18430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "484444f270d74131b2e7ef1d270409e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484444f270d74131b2e7ef1d270409e4" member_type: VOTER last_known_addr { host: "127.27.75.254" port: 41241 } } peers { permanent_uuid: "946b11efc5be47a191e3a5f1303831dd" member_type: VOTER last_known_addr { host: "127.27.75.253" port: 41381 } } peers { permanent_uuid: "58279e179f424e429fb72bfaba7d044e" member_type: VOTER last_known_addr { host: "127.27.75.252" port: 41481 } } }
I20260923 18:15:24.756228 18430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd [sys.catalog]: This master's current role is: FOLLOWER
I20260923 18:15:24.778934 18449 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 18:15:24.780326 18449 catalog_manager.cc:1504] Loading token signing keys...
I20260923 18:15:24.795127 18449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 484444f270d74131b2e7ef1d270409e4: Generated new TSK 0
I20260923 18:15:24.796151 18449 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 18:15:24.824115 18414 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:24.820870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.27.75.254:41241)
0923 18:15:24.821006 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.821010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.822975 (+ 1965us) server_negotiation.cc:407] Connection header received
0923 18:15:24.823178 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.823182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.823236 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.823288 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.823976 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59000: BlockingRecv error: recv got EOF from 127.0.0.1:59000 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 18:15:24.826192 18411 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:24.822351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.27.75.252:41481)
0923 18:15:24.822464 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.822468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.824849 (+ 2381us) server_negotiation.cc:407] Connection header received
0923 18:15:24.825105 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.825108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.825200 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.825251 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.826060 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59644: BlockingRecv error: recv got EOF from 127.0.0.1:59644 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 18:15:24.826989 18335 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:24.821705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.27.75.253:41381)
0923 18:15:24.821815 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.821819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.824634 (+ 2815us) server_negotiation.cc:407] Connection header received
0923 18:15:24.825389 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.825393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.825444 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.825492 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.826872 (+ 1380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43192: BlockingRecv error: recv got EOF from 127.0.0.1:43192 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260923 18:15:24.831763 18335 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:24.830218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.27.75.253:41381)
0923 18:15:24.830328 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.830332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.830786 (+ 454us) server_negotiation.cc:407] Connection header received
0923 18:15:24.831023 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.831026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.831105 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.831157 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.831654 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43208: BlockingRecv error: recv got EOF from 127.0.0.1:43208 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260923 18:15:24.837000 18335 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:24.835438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.27.75.253:41381)
0923 18:15:24.835549 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.835553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.836074 (+ 521us) server_negotiation.cc:407] Connection header received
0923 18:15:24.836262 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.836265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.836318 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.836369 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.836886 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43214: BlockingRecv error: recv got EOF from 127.0.0.1:43214 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260923 18:15:24.843020 18335 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:24.841359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.27.75.253:41381)
0923 18:15:24.841468 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.841472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.842037 (+ 565us) server_negotiation.cc:407] Connection header received
0923 18:15:24.842239 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.842243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.842295 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.842346 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.842897 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43216: BlockingRecv error: recv got EOF from 127.0.0.1:43216 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260923 18:15:24.851442 18335 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:24.849609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.27.75.253:41381)
0923 18:15:24.849755 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.849760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.850386 (+ 626us) server_negotiation.cc:407] Connection header received
0923 18:15:24.850607 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.850611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.850677 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.850730 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.851314 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43228: BlockingRecv error: recv got EOF from 127.0.0.1:43228 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 18:15:24.864176 18335 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:24.862335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.27.75.253:41381)
0923 18:15:24.862463 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.862468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.863096 (+ 628us) server_negotiation.cc:407] Connection header received
0923 18:15:24.863307 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.863310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.863371 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.863427 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.864046 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43232: BlockingRecv error: recv got EOF from 127.0.0.1:43232 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 18:15:24.891947 18335 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:24.888703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.27.75.253:41381)
0923 18:15:24.888850 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.888855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.890980 (+ 2125us) server_negotiation.cc:407] Connection header received
0923 18:15:24.891180 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.891184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.891236 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.891286 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.891815 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43236: BlockingRecv error: recv got EOF from 127.0.0.1:43236 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260923 18:15:24.927582 18335 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:24.926304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.27.75.253:41381)
0923 18:15:24.926435 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.926439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.926666 (+ 227us) server_negotiation.cc:407] Connection header received
0923 18:15:24.926793 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.926797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.926855 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.926925 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.927468 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43250: BlockingRecv error: recv got EOF from 127.0.0.1:43250 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 18:15:24.995128 18335 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:24.994024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.27.75.253:41381)
0923 18:15:24.994173 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:24.994177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:24.994269 (+ 92us) server_negotiation.cc:407] Connection header received
0923 18:15:24.994442 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:24.994445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:24.994501 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:24.994587 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:24.995027 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43266: BlockingRecv error: recv got EOF from 127.0.0.1:43266 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 18:15:25.126442 18335 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:25.125308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.27.75.253:41381)
0923 18:15:25.125435 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:25.125438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:25.125525 (+ 87us) server_negotiation.cc:407] Connection header received
0923 18:15:25.125697 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:25.125700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:25.125762 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:25.125846 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:25.126335 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43272: BlockingRecv error: recv got EOF from 127.0.0.1:43272 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 18:15:25.385967 18335 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 18:15:25.384719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.27.75.253:41381)
0923 18:15:25.384848 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:25.384860 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:25.385014 (+ 154us) server_negotiation.cc:407] Connection header received
0923 18:15:25.385187 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:25.385190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:25.385240 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:25.385311 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 18:15:25.385822 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43276: BlockingRecv error: recv got EOF from 127.0.0.1:43276 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 18:15:25.696211 18441 catalog_manager.cc:1259] Loaded cluster ID: 85eb06f525764e3d8434283babd0f32a
I20260923 18:15:25.696312 18441 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd: loading cluster ID for follower catalog manager: success
I20260923 18:15:25.698175 18441 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd: acquiring CA information for follower catalog manager: success
I20260923 18:15:25.699486 18441 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 946b11efc5be47a191e3a5f1303831dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:15:25.707301 18462 catalog_manager.cc:1259] Loaded cluster ID: 85eb06f525764e3d8434283babd0f32a
I20260923 18:15:25.707393 18462 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e: loading cluster ID for follower catalog manager: success
I20260923 18:15:25.709196 18462 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e: acquiring CA information for follower catalog manager: success
I20260923 18:15:25.710501 18462 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 58279e179f424e429fb72bfaba7d044e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 18:15:25.904518 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:25.900173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.27.75.253:41381)
0923 18:15:25.900398 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:25.900401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:25.900415 (+ 14us) server_negotiation.cc:407] Connection header received
0923 18:15:25.900639 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:25.900643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:25.900774 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:25.900889 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:15:25.901257 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:25.901859 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:25.902817 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:25.903057 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:25.903144 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:25.903547 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:15:25.904060 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:15:25.904159 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:25.904289 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 18:15:25.904353 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260923 18:15:25.911769 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:25.908073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.27.75.252:41481)
0923 18:15:25.908308 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:25.908311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:25.908326 (+ 15us) server_negotiation.cc:407] Connection header received
0923 18:15:25.908408 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:25.908410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:25.908532 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:25.908630 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:15:25.908851 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:25.909434 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:25.910340 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:25.910569 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:25.910652 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:25.910962 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:15:25.911412 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:15:25.911502 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:25.911589 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 18:15:25.911641 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260923 18:15:25.918987 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:25.915517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.27.75.252:41481)
0923 18:15:25.915624 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:25.915628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:25.915641 (+ 13us) server_negotiation.cc:407] Connection header received
0923 18:15:25.915684 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:25.915687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:25.915730 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:25.915792 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:15:25.916077 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:25.916824 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:25.917771 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:25.918099 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:25.918192 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:25.918527 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:15:25.918612 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:15:25.918676 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:25.918782 (+ 106us) server_negotiation.cc:299] Negotiation successful
0923 18:15:25.918840 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 18:15:25.919215 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:25.914884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.27.75.253:41381)
0923 18:15:25.915012 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:25.915016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:25.915229 (+ 213us) server_negotiation.cc:407] Connection header received
0923 18:15:25.915322 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:25.915325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:25.915395 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:25.915472 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:15:25.915839 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:25.916453 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:25.917855 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:25.918187 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:25.918273 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:25.918547 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:15:25.918618 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:15:25.918681 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:25.919034 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 18:15:25.919088 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 18:15:25.919478 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 18:15:25.914847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.27.75.254:41241)
0923 18:15:25.915105 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 18:15:25.915109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 18:15:25.915141 (+ 32us) server_negotiation.cc:407] Connection header received
0923 18:15:25.915373 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 18:15:25.915377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 18:15:25.915545 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 18:15:25.915676 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 18:15:25.915846 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:25.916635 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:25.917836 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 18:15:25.918088 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 18:15:25.918179 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 18:15:25.918487 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 18:15:25.918852 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 18:15:25.918946 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 18:15:25.919262 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 18:15:25.919337 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":61,"threads_started":1}
I20260923 18:15:25.922600 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 18187
I20260923 18:15:25.928298 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 18262
I20260923 18:15:25.933351 27951 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknO4ss8/build/debug/bin/kudu with pid 18337
2026-09-23T18:15:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3052 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6107 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283437 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST